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L9: Entry 1 of 1 



File: USPT 



Nov 5, 2002 



DOCUMENT- IDENTIFIER: US 6477550 Bl 

TITLE: Method and system for processing events related to a first type of browser from 
a second type of browser 

Application Filing Date (1) : 
19990316 

Detailed Description Text (17) : 

When a client computer displays a web page, a user may use an attached input device 
such as a mouse or a key board to instruct the browser to perform further actions such 
as accessing other web pages for which links are provided in the displayed web page, or 
download and play an audio or video stream embedded in the web page. These actions 
performed by a user create what are known in the client -server computing jargon as 
"events," which can be acted upon by code in a computer, if the computer is configured 
to do so. 
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L10 : Entry 2 of 26 File: PGPB Dec 13, 2001 



DOCUMENT- IDENTIFIER : US 20010052001 Al 
TITLE: DIGITAL DEPARTMENT SYSTEM 



Abstract Paragraph (1) : 

A digital department system is disclosed. The digital department system of the present 
invention includes a network management center, a network operating center that is 
coupled to the network management center, a multimedia server, a multicasting 
transmission medium coupling the network operating center and the multimedia server, a 
listening post coupled to the multimedia server by the network, and one or more 
audio/ video display capable of displaying video and capable of playing audio, the audio 
video display coupled to the multimedia server by the network. The audio and video 
content are stored in digitized files on the multimedia server for distribution 
throughout the site via the network. 

Application Filing Date (1) : 
19980807 

Summary of Invention Paragraph (13): 

[0011] In one embodiment of the present invention, a digital department system is 
provided. The digital department system of the present invention includes a network 
management center, a network operating center that is coupled to the network management 
center, a multimedia server, a multicasting transmission medium coupling the network 
operating center and the multimedia server, a listening post coupled to the multimedia 
server by the network, and one or more audio/video display capable of displaying video 
and capable of playing audio, the audio video display coupled to the multimedia server 
by the network. The audio and video content are stored in digitized files on the 
multimedia server for distribution throughout the site via the network. 

Detail Description Paragraph (12) : 

[0028] NMC 110 maintains a catalog (database) of products for preview. NMC 110, in 
accumulating and compiling this information, also digitizes this information and 
provides it to a Network Operations Center (NOC) 120 in the form of digitized data 
files 122. It will be noted that data files 122, although referred to in terms of 
digitized audiovisual content, can also be streaming audio, streaming video, or other 
such information. Each product may have associated digital files containing information 
pertaining to the product. Alternatively, all the information may be compiled into one 
file. Following are examples of the types of files/information that may be catalogued 
and maintained: 

Detail Description Paragraph (76) : 

[0092] One example of a multicasting technique is the Multicast File Transfer Protocol 
(MFTP) from Starburst . TM . . This protocol is described in great detail in the 
specification entitle "STARBURST MULTICAST FILE TRANSFER PROTOCOL (MFTP) 
SPECIFICATION," (filename: draft-miller-mftp-spec-03.txt; dated April, 1998) which can 
be viewed at the time of this writing at the following universal resource locator on 
the World Wide Web : 

Detail Description Paragraph (90) : 

[0105] Receiver /decoder 135 is capable of receiving, processing, and providing voice, 
video, data, and other forms of information to various devices within commercial sales 
outlet 130. While the configuration of the communications network 125 (and in 
particular, transmitting station 121, receiving station 128, and receiver/decoder 135) 
will vary according to the technology used to distribute digital data files 122. For 
example, while transmitting station 121 and receiving station 128 are described in 
terms of a satellite network, one of skill in the art will recognize that these 
elements could support broadcast or duplex communications systems. These elements could 
therefore be satellite transmitters/receiver pairs, a multicast network, a UUCP 
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(Unix-to-Unix CoPy) network, or the like. Alternatively, these elements could be 
network interface cards, microwave transceivers, infrared transceivers, or the like. In 
this example, a satellite broadcasting system is employed. Transmitting station 121 and 
satellite 127 are implemented using commercially available satellite communication 
technology, and so are not described in detail herein. 

Detail Description Paragraph (129) : 

[0144] Receiver/decoder 135 is preferably capable of receiving both digital and analog 
information. With regard to data reception and distribution, receiver/decoder 135 is 
connected to various network nodes in commercial sales outlet 130 via a network system 
150. Network 150, while it may be directly connected to various other nodes in 
commercial sales outlet 130 is connected in FIG. IB to multimedia server 160, as noted. 
For most of data files 122, once they are received by receiving station 128 at 
commercial sales outlet 130, they are passed to receiver/decoder 135, which in turn 
passes them on to multimedia server 160. Multimedia server 160 is connected via the 
network (e.g., an ether network using a TCP/IP protocol stack, and using FTP file 
transfers to distribute the promotional information) to various nodes in commercial 
sales outlet 130. These nodes include (but are not limited to) an in-line home video 
station 161, an in-line computer hardware and software interactive display 165, an 
in-store radio system 170, a wall-of-eyes 180 (which may be connected to either 
multimedia server 160, or to receiver /decoder 135 via a video switch 175, which is 
optional), listening posts 185, audio/video endcaps 190, audio/video endcaps in other 
departments 195, on-line services 200 and an entity LAN 210. Entity LAN 210 may be 
connected to an on-line internet commerce access system 220, a UNIX server 230 and/or a 
CBL server 240, among other such possible connections. 

Detail Description Paragraph (309) : 

[0324] As noted, data sent via communications network 125 can be of several types. For 
example, receiver /decoder 135 may provide audiovisual training information to video 
cassette recorder (VCR) 140 via the direct video capabilities that communications 
network 125 may support (e.g., real-time analog, high-definition television (HDTV), or 
digital video information) . Training VCR 140 allows such audiovisual information to be 
recorded for later playback, to allow the audiovisual information to be replayed for 
off-line training or during multiple training sessions. 
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L10: Entry 3 of 26 



File: PGPB 



Aug 9, 2001 



DOCUMENT- IDENTIFIER: US 20010013068 Al 

TITLE: INTERLEAVED MULTIPLE MULTIMEDIA STREAM FOR SYNCHRONIZED TRANSMISSION OVER A 
COMPUTER NETWORK 



Abstract Paragraph (1) : 

The production of an interleaved multimedia stream for servers and client computers 
coupled to each other by a diverse computer network which includes local area networks 
(LANs) and/or wide area networks (WANs) such as the internet. Interleaved multimedia 
streams can include compressed video frames for display in a video window, accompanying 
compressed audio frames and annotation frames. In one embodiment, a producer captures 
separate video/audio frames and generates an interleaved multimedia file. In another 
embodiment, the interleaved file include annotation frames which provide either 
pointer (s) to the event (s) of interest or include displayable data embedded within the 
annotation stream. The interleaved file is then stored in the web server for subsequent 
retrieval by client computer(s) in a coordinated manner, so that the client computer(s) 
is able to synchronously display the video frames and displayable event (s) in a video 
window and event window ( s) , respectively. In some embodiments, the interleaved file 
includes packets with variable length fields, each of which are at least one numerical 
unit in length. 

Application Filing Date (1) : 
19970325 

Cross Reference to Related Applications Paragraph (1) : 

[0001] Pending U.S. patent application Ser. No. , entitled "Production of a Video 

Stream with Synchronized Annotations over a Computer Network", Attorney Docket Number 
VXT. sub. --703, assigned to VXtreme, Inc. and filed Mar. 14, 1997, is herein 
incorporated by reference in its entirety. 

Summary of Invention Paragraph (6) : 

[0006] Pending patent application VXT . sub -703 describes the production of separate 
video, audio and annotation streams for synchronous delivery from a stream server to a 
client computer. However, if the stream server is not available or not affordable to 
the end user at the client computer, then the client computer may only have access to 
wep servers which are not designed to provide synchronous delivery capability of the 
separate video, audio and annotation streams. 

Summary of Invention Paragraph (7) : 

[0007] In view of the foregoing, there are desired techniques for generating integrated 
multimedia content such as video and audio frames, for synchronous delivery from a web 
server client computer(s). 

Summary of Invention Paragraph (10) : 

[0009] In one embodiment, a producer captures separate video/audio frames and generates 
an interleaved multimedia file. In another embodiment, the interleaved file include 
annotation frames which provide either pointer (s) to the event (s) of interest or 
include displayable data embedded within the annotation stream. Accordingly, each 
annotation frame includes either an event locator or an event data. In addition, each 
annotation frame includes an event time marker which corresponds to the time stamp (s) 
of associated video frame (s) within the video stream . 

Summary of Invention Paragraph (11) : 

[0010] The interleaved file is then stored in the web server for subsequent retrieval 
by client computer (s) in a coordinated manner, so that the client computer (s) is able 
to synchronously display the video frames and displayable event (s) in a video window 
and event window(s), respectively. In some embodiments, the interleaved file includes 
packets with variable length fields, each of which are at least one numerical unit in 
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Brief Description of Drawings Paragraph (6) : 

[0016] FIG. 4A is a flowchart illustrating the capture of a live video/audio stream 
from a video camera or from a previously stored video file. 

Brief Description of Drawings Paragraph (8) : 

[0018] FIG. 5 shows an exemplary format for storing and delivering a compressed video 
stream . 

Brief Description of Drawings Paragraph (12) : 

[0022] FIG. 9A illustrates one embodiment of the client computer which includes a web 
browser and a browser plug-in module for interfacing a web browser with a client 
module . 

Brief Description of Drawings Paragraph (13) : 

[0023] FIG. 9B illustrates another embodiment of the client computer in which the 
browser plug-in module receives an interleaved stream from the web server and 
distributes the video/audio stream(s) to the video/audio decoder (s) and the annotation 
stream(s) to the annotation interpreter. 

Detail Description Paragraph (13) : 

[0042] FIG. 2 is a block diagram showing an exemplary hardware environment for 
practicing the annotated video -on -demand (VOD) system of the present invention. The VOD 
system includes a production station 210, a stream server 220, at least one web server 
230 and at least one client computer 240, each of which can be implemented using 
computer system 10 0 described above. Stream server 22 0 and web server 23 0 are coupled 
to client computer 240 via a computer network 290, e.g., the internet. Note that the 
disclosed hardware environment is exemplary. For example, production station 210 and 
stream server 220 can be implemented using two separate computer systems or using one 
computer system. In addition, if production station 210 and stream server 220 are 
implemented on separate computer systems as shown in FIG. 2, an optional direct 
connection (not shown) between production station 210 and stream server 220 can provide 
faster uploads of compressed video and annotation streams. In the following 
description, an audio stream optionally accompanies each video stream . 

Detail Description Paragraph (14) : 

[0043] A producer 215, installed in production station 210, is a user-friendly tool for 
use by a designer 219 to create a synchronization script which includes annotation 
stream ( s) . The annotation stream(s) define the content(s) of a LiveScreen display 245 
to be displayed on client computer 240 for a viewer 249. LiveScreen 245 display 
provides a graphical user interface (GUI) with multiple windows for synchronously 
displaying a video stream from stream server 220 and at least one displayable event 
stream. Examples of displayable events include textual/graphical information such as 
HTML-scripted web page(s) from web server 230. 

Detail Description Paragraph (15) : 

[0044] In one embodiment, as shown in FIG. 3A, producer 215a includes a capture module 
317a and an author module 318a. Production station 210 includes 16 MB of RAM and a 1 GB 
hard disk drive for capturing and storing an uncompressed or precompressed video 
stream . Sources for generating video streams include a video camera 312, a video 
cassette recorder (VCR) (not shown) or a previously digitized video file 314, e.g. , a 
Video for Windows (.avi) file. For ease of installation and use by designer 219, 
producer 215a is implemented in a host environment which includes a window-based 
operating system such as Microsoft Windows 95 and a web browser such as Netscape's 
Navigator 3.x. 

Detail Description Paragraph (16) : 

[0045] Referring also to the flowchart of FIG. 4A, in step 410 capture module 317a 
captures a live video/audio stream from video camera 312 or from the previously stored 
video file 314. If video camera 312 provides an analog video stream, e.g., an NTSC 
signal, a hardware capture card (not shown) provides the required conversion from the 
analog video stream to a digitized video stream . Because temporary storage of 
uncompressed video data is memory intensive, some form of pre-compression can be used 
to reduce the memory storage requirement of the input video stream during capture step 
410 and prior to compression step 420. 

Detail Description Paragraph (17) : 

[0046] In step 420, capture module 420 compresses the digitized video stream using a 
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suitable compression technique. In this embodiment, depending on the bandwidth capacity 
of the connection provided by network 2 90 between stream server 220 and client computer 
240, e.g., a POTS modem, ISDN or Ethernet, a suitable frame resolution and frame rate 
combination is selected. FIG. 5 shows an exemplary format 50 0 for storing and 
delivering a compressed video stream . 

Detail Description Paragraph (21) : 

[0050] Designer 219 may view frames from video stream 500 displayed in video window 720 
for referencing and selecting appropriate time stamps to use in generating annotation 
streams. Within video window 72 0, VCR function buttons, e.g., a rewind button 724, a 
play button 726 and a fast forward button 728, are available for designer 219 to 
quickly traverse video stream 500. Since video window 720 is provided as a convenience 
for designer 219, if designer 219 has prior knowledge of the content of the video 
stream, designer 219 may proceed with the generation of the annotation streams without 
viewing video window 720. 

Detail Description Paragraph (27) : 

[0056] In accordance with another aspect of the invention, LiveScreen display 600 also 
includes a table of contents (TOC) 630, enabling viewer 249 at client computer 240 to 
skip forward or backward to a point within the entire video/audio stream 500. TOC 630 
include one or more content labels, each indexed to a corresponding time stamp in video 
stream 500, as defined by TOC time markers 791, 792, 793, 794 in LiveScreen display 
600 . 

Detail Description Paragraph (28) : 

[0057] Referring now to FIG . 9A, in one embodiment of the present invention, client 
computer 240 includes a web browser 950 and a browser plug-in module 952a for 
interfacing web browser 950 with a main client module 960. Client module 960 includes 
an event registry 962, playout buffer(s) 966, video/audio decoder (s) 964, video /audio 
renderer(s) 965 and one or more dynamically loadable event applet(s), e.g., flipper 
applet 967, ticker applet 968 and VCR applet 969. In this embodiment, event registry 
962 also functions as an annotation interpreter 963 . 

Detail Description Paragraph (29) : 

[0058] FIG. 10A is a flowchart illustrating the operation of client module 960. Assume 
that viewer 249 has not previously loaded client module 960 in client computer 240, but 
has already loaded a web browser 950, e.g., Netscape's Navigator (step 1010). Viewer 
249 surfs the world-wide web (www) via the internet and locates a web site of interest 
to viewer 249. Typically, the web site of interest is hosted on web server 230. 
Accordingly, a target web page is downloaded from web server 230 and displayed on 
client computer 240. 

Detail Description Paragraph (30) : 

[0059] The target web page includes a link to a customized LiveScreen display, e.g., 
display 600. If client module 960 has not been previously loaded, client module 960 is 
now loaded over web browser 950 for processing video/audio and annotation streams (step 
1020) . Depending on the implementation, a copy of client module 960 may be available 
from the web site of interest. Alternatively, the target web page may provide a HTML 
link to another web server which has an updated copy of client module 960. 

Detail Description Paragraph (31) : 

[0060] Referring now to FIG. 10B, first, browser plug-in module 952a is installed over 
web browser 950 (step 1022) . As discussed above, plug-in module 952a provides the 
interface between client module 960 and web browser 950. The target web page provides a 
HTML link to the format for LiveScreen display 600. The LiveScreen display format is 
retrieved and display 600 is installed on client computer 240 using web browser 950 

(step 1024) . 

Detail Description Paragraph (33) : 

[0062] Referring back to FIG. 10A, encoded video/audio frames and associated annotation 
frames are streamed from stream server 220 to client computer 240 for synchronous 
display (step 1030) . Streaming video and audio streams over a network is very efficient 
because streaming eliminates the need for a large buffer at client computer 240. In 
addition, streaming also provides flexibility, e.g., switching video sources midstream 
is possible without wasting network resources since streaming is based on a pseudo 
just-in-time (JIT) protocol and does not involve downloads of the entire video stream 
prior to display at client computer 240. If the underlying transmission protocol is 
HTTP, then video, audio and annotation packets are initially "pulled" by client 
computer 240 from server 220 using HTTP "get" packet (s). 
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Detail Description Paragraph (34) : 

[0063] Next, the encoded video/audio streams are decoded by decoder 964, i.e., 
decompressed using a suitable technique, and then displayed at client computer 240 by 
renderer 965 (step 1040) . 

Detail Description Paragraph (37) : 

[00 66] Further, since the video and annotation streams are handled synchronously but 
separately by video decoder 964 and annotation interpreter 963, respectively, steps 
1040 and 1050 can occur concurrently or consecutively. As discussed above, event 
registry 962 is capable of dynamic registration of event applets. Accordingly, 
annotation interpreter 963 is adaptable, and capable of automatic installation and 
linking of new event applet (s) to add new class (es) of displayable events for client 
computer 240 . 

Detail Description Paragraph (38) : 

[0067] After registering with event registry 962, flipper applet 967 provides the 
location of the flipper stream to broswer 950 which then begin receiving the flipper 
stream from stream server 220. Flipper annotation frames are provided by stream server 
220 synchronously with the video/audio frames to client module 960 so that the 
annotations, i.e., displayable events can be synchronized for display at client 
computer 240 (step 1060) . In this example, URL addresses, for synchronizing HTML page 
flips with video stream are provided to web broswer 950 thereby permitting client 
computer 240 to subsequently retrieve and display various textual and graphical 
elements changing at predetermined points corresponding to the timeline of the video 
stream . Note that HTML pages can be retrieved from one or more web server (s) 230~! 

Detail Description Paragraph (39) : 

[0068] Similarly, after registering with event registry 962, ticker (tape) applet 968 
provides the location of the ticker stream to broswer 950 which then begins receiving 
the ticker stream from stream server 220. Ticker annotation frames are provided by 
stream server 220 synchronously with the video/audio frames so that the annotations, 
i.e., displayable ticker data can be synchronized for display at client computer 240 at 
predetermined points corresponding to the timeline of the video stream . 

Detail Description Paragraph (42) : 

[0071] As shown in FIG. 11, a table of content 630 with content labels enables viewer 
249 to skip forward or backward to predetermined locations in the video/audio stream. 
First, viewer 249 selects a content label of interest (step 1110) . Examples of suitable 
content labels are section headings of the video stream . Next, client module 960 sends 
a message to stream server 22 0 with the time stamp of an I -frame from the video stream 
whose location is close to selected content label (step 1120) . In this embodiment, an 
I -frame is a video frame which includes data for a complete video frame. Although 
computationally more intensive, it is also possible to select a P-frame and then 
reconstructed a complete video starting from a neighboring I -frame close to the 
selected P-frame. 



Detail Description Paragraph (44) : 

[0073] Referring now to FIGS. 3B and 9B, in another embodiment, instead of streaming 
three separate video, audio and annotation streams from stream server 220 to client 
computer 240, an interleaved video/audio/annotation file is produced by producer 215b, 
stored in web server 230, and subsequently provided to client module 960 on demand via 
web browser 950. Note that an interleaved file can include any two or more frame types, 
e.g., video and audio frames, video and annotation frames, or audio and annotation 
frames . 



Detail Description Paragraph (45) : 

[0074] Advantages of this embodiment include simplified synchronous delivery of video, 
audio and annotation frames to client computer 240. Simplicity is accomplished by 
eliminating the need for stream server 220, whose primary function is to manage the 
transmission of several separate video, audio and annotation streams from stream server 
220 to client computer 240. In this embodiment, since all the video, audio and 
annotation frames are combined into a single interleaved stream and are pre-sorted by 
timestamp values, the interleaved stream can now be stored in web server 230 and 
delivered in the form of HTTP data. 



Detail Description Paragraph (52) : 

[0081] In accordance with another aspect of this embodiment, the data packets 1320, 
1330 ' • • • 1 3 90 for streaming video and audio frames include a variable packet length 
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field 1324 as shown in FIGS 



exemplary formats 1324a, 1324b, and 1324c of the variable packet length field 1324 are 
shown. In this implementation, the length of the variable packet length field is in 
multiples of number units. For example, formats 1324a, 1324b and 1324c can be one 
numerical unit in length, three numerical units in length and seven numerical units in 
length, respectively. As is known to one skilled in the art, regardless of the size of 
the packet length field, the packet length can be represented by a number of different 
methods, such as simple binary, one's complement, BCD and floating point. 



1. In a computer having a processor and memory, said computer useful in association 
with a web server coupled to a client computer via a network, a method for producing an 
interleaved multimedia file from a video file and an audio file, the method comprising 
the steps of: if a video frame buffer is empty, then retrieving a first video frame 
from the video file, said first video frame including a video timestamp; if an audio 
frame buffer is empty, then retrieving a first audio frame from the audio file, said 
first audio frame including an audio timestamp; and if the video timestamp is less than 
or equal to the audio timestamp, then writing the first video frame to a first packet 
of the interleaved file; and retrieving a second video frame from the video file; else 
if the audio timestamp is less than or equal to the video timestamp, then writing the 
first audio frame to a second packet of the interleaved file; and retrieving a second 
audio frame from the audio file. 



CLAIMS: 



5 of 5 



9/12/03 9:16 PM 



Record Display Form 




L10 : Entry 4 of 26 File: USPT Apr 22, 2003 



DOCUMENT- IDENTIFIER: US 6553404 B2 
TITLE: Digital system 



Abstract Text (1) : 

A digital department system is disclosed. The digital department system includes a 
network management center, a network operating center that is coupled to the network 
management center, a multimedia server, a multicasting transmission medium coupling the 
network operating center and the multimedia server, a listening post coupled to the 
multimedia server by the network, and one or more audio/video display capable of 
displaying video and capable of playing audio, the audio video display coupled to the 
multimedia server by the network. The audio and video content are stored in digitized 
files on the multimedia server for distribution throughout the site via the network. 

Application Filing Date (1) : 
19980807 

Brief Summary Text (13) : 

In one embodiment of the present invention, a digital department system is provided. 
The digital department system of the present invention includes a network management 
center, a network operating center that is coupled to the network management center, a 
multimedia server, a multicasting transmission medium coupling the network operating 
center and the multimedia server, a listening post coupled to the multimedia server by 
the network, and one or more audio/video display capable of displaying video and 
capable of playing audio, the audio video display coupled to the multimedia server by 
the network. The audio and video content are stored in digitized files on the 
multimedia server for distribution throughout the site via the network. 

Detailed Description Text (12) : 

NMC 110 maintains a catalog (database) of products for preview. NMC 110, in 
accumulating and compiling this information, also digitizes this information and 
provides it to a Network Operations Center (NOC) 120 in the form of digitized data 
files 122. It will be noted that data files 122, although referred to in terms of 
digitized audiovisual content, can also be streaming audio, streaming video, or other 
such information. Each product may have associated digital files containing information 
pertaining to the product. Alternatively, all the information may be compiled into one 
file. Following are examples of the types of files/information that may be catalogued 
and maintained: Video clips (music video, movie clip, product demonstration, and the 
like) Liner notes Date for broadcast Audio clips Song titles Region for broadcast 
Graphics Album/artist info Date to delete 

Detailed Description Text (33) : 

One example of a multicasting technique is the Multicast File Transfer Protocol (MFTP) 
from Starburst .TM. . This protocol is described in great detail in the specification 
entitle "STARBURST MULTICAST FILE TRANSFER PROTOCOL (MFTP) SPECIFICATION," (filename: 
draft-miller-mftp-spec-03.txt; dated April, 1998) which can be viewed at the time of 
this writing at the following universal resource locator on the World Wide Web : 

Detailed Description Text (46) : 

Receiver /decoder 135 is capable of receiving, processing, and providing voice, video, 
data, and other forms of information to various devices within commercial sales outlet 
130. While the configuration of the communications network 125 (and in particular, 
transmitting station 126, receiving station 128, and receiver/decoder 135) will vary 
according to the technology used to distribute digital data files 122. For example, 
while transmitting station 126 and receiving station 128 are described in terms of' a 
satellite network, one of skill in the art will recognize that these elements could 
support broadcast or duplex communications systems. These elements could therefore be 
satellite transmitters/receiver pairs, a multicast network, a UUCP (Unix- to-Unix CoPy) 
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network, or the like. Alternatively, these elements could be network interface cards, 
microwave transceivers, infrared transceivers, or the like. In this example, a 
satellite broadcasting system is employed. Transmitting station 126 and satellite 127 
are implemented using commercially available satellite communication technology, and so 
are not described in detail herein. 

Detailed Description Text (65) : 

Receiver/decoder 13 5 is preferably capable of receiving both digital and analog 
information. With regard to data reception and distribution, receiver/decoder 135 is 
connected to various network nodes in commercial sales outlet 130 via a network system 
150. Network 150, while it may be directly connected to various other nodes in 
commercial sales outlet 130 is connected in FIG. IB to multimedia server 160, as noted. 
For most of data files 122, once they are received by receiving station 128 at 
commercial sales outlet 130, they are passed to receiver/decoder 135, which in turn 
passes them on to multimedia server 160. Multimedia server 160 is connected via the 
network (e.g., an ether network using a TCP/IP protocol stack, and using FTP file 
transfers to distribute the promotional information) to various nodes in commercial 
sales outlet 130. These nodes include (but are not limited to) an in-line home video 
station 161, an in-line computer hardware and software interactive display 165, an 
in-store radio system 170, a wall -of -eyes 180 (which may be connected to either 
multimedia server 160, or to receiver /decoder 13 5 via a video switch 175, which is 
optional), listening posts 185, audio/video endcaps 190, audio/video endcaps in other 
departments 195, on-line services 200 and an entity LAN 210. Entity LAN 210 may be 
connected to an on-line internet commerce access system 220, a UNIX server 230 and/or a 
CBL server 240, among other such possible connections. 

Detailed Description Text (149) : 

As noted, data sent via communications network 125 can be of several types. For 
example, receiver /decoder 135 may provide audiovisual training information to video 
cassette recorder (VCR) 140 via the direct video capabilities that communications 
network 125 may support (e.g., real-time analog, high-definition television (HDTV), or 
digital video information) . Training VCR 140 allows such audiovisual information to be 
recorded for later playback, to allow the audiovisual information to be replayed for 
off-line training or during multiple training sessions. 
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DOCUMENT- IDENTIFIER: US 6449653 B2 

** See image for Certificate of Correction ** 

TITLE: Interleaved multiple multimedia stream for synchronized transmission over a 
computer network 



Abstract Text (1) : 

The production of an interleaved multimedia stream for servers and client computers 
coupled to each other by a diverse computer network which includes local area networks 
(LANs) and/or wide area networks (WANs) such as the internet. Interleaved multimedia 
streams can include compressed video frames for display in a video window, accompanying 
compressed audio frames and annotation frames. In one embodiment, a producer captures 
separate video/audio frames and generates an interleaved multimedia file. In another 
embodiment, the interleaved file include annotation frames which provide either 
pointer (s) to the event (s) of interest or include displayable data embedded within the 
annotation stream. The interleaved file is then stored in the web server for subsequent 
retrieval by client computer (s) in a coordinated manner, so that the client computer (s) 
is able to synchronously display the video frames and displayable event (s) in a video 
window and event window(s), respectively. In some embodiments, the interleaved file 
includes packets with variable length fields, each of which are at least one numerical 
unit in length. 

Application Filing Date (1) : 
19970325 

Parent Case Text (2) : 

Pending U.S. patent application Ser. No. 08/818,804, entitled "Production of a Video 
Stream with Synchronized Annotations over a Computer Network" , Attorney Docket Number 
VXT.sub.-- 703, assigned to VXtreme, Inc. and filed Mar. 14, 1997, is herein 
incorporated by reference in its entirety. Patent application Ser. No. 08/818,804 
issued as U.S. Pat. No. 6,006,241 on Dec. 21, 1999, assignee Microsoft Corporation of 
Redmond, Wash. 

Brief Summary Text (6) : 

Pending patent application VXT.sub.-- 703 describes the production of separate video, 
audio and annotation streams for synchronous delivery from a stream server to a client 
computer. However, if the stream server is not available or not affordable to the end 
user at the client computer, then the client computer may only have access to web 
servers which are not designed to provide synchronous delivery capability of the 
separate video, audio and annotation streams. 

Brief Summary Text (7) : 

In view of the foregoing, there are desired techniques for generating integrated 
multimedia content such as video and audio frames, for synchronous delivery from a web 
server client computer(s). 

Brief Summary Text (10) : 

In one embodiment, a producer captures separate video/audio frames and generates an 
interleaved multimedia file. In another embodiment, the interleaved file include 
annotation frames which provide either pointer (s) to the event (s) of interest or 
include displayable data embedded within the annotation stream. Accordingly, each 
annotation frame includes either an event locator or an event data. In addition, each 
annotation frame includes an event time marker which corresponds to the time stamp (s) 
of associated video frame (s) within the video stream . 

Brief Summary Text (11) : 

The interleaved file is then stored in the web server for subsequent retrieval by 
client computer (s) in a coordinated manner, so that the client computer (s) is able to 
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synchronously display the video frames and displayable event (s) in a video window and 
event window (s) , respectively. In some embodiments, the interleaved file includes 
packets with variable length fields, each of which are at least one numerical unit in 
length. 

Drawing Description Text (6) : 

FIG. 4A is a flowchart illustrating the capture of a live video/audio stream from a 
video camera or from a previously stored video file. 

Drawing Description Text (8) : 

FIG. 5 shows an exemplary format for storing and delivering a compressed video stream . 
Drawing Description Text (12) : 

FIG. 9A illustrates one embodiment of the client computer which includes a web browser 
and a browser plug-in module for interfacing a web browser with a client module . 

Drawing Description Text (13) : 

FIG. 9B illustrates another embodiment of the client computer in which the browser 
plug-in module receives an interleaved stream from the web server and distributes the 
video/audio stream (s) to the video/audio decoder (s) and the annotation stream(s) to the 
annotation interpreter. 

Detailed Description Text (13) : 

FIG. 2 is a block diagram showing an exemplary hardware environment for practicing the 
annotated video -on-demand (VOD) system of the present invention. The VOD system 
includes a production station 210, a stream server 220, at least one web server 230 and 
at least one client computer 240, each of which can be implemented using computer 
system 100 described above. Stream server 220 and web server 230 are coupled to client 
computer 240 via a computer network 290, e.g., the internet. Note that the disclosed 
hardware environment is exemplary. For example, production station 210 and stream 
server 220 can be implemented using two separate computer systems or using one computer 
system. In addition, if production station 210 and stream server 220 are implemented on 
separate computer systems as shown in FIG. 2, an optional direct connection (not shown) 
between production station 210 and stream server 220 can provide faster uploads of 
compressed video and annotation streams. In the following description, an audio stream 
optionally accompanies each video stream . 

Detailed Description Text (14) : 

A producer 215, installed in production station 210, is a user-friendly tool for use by 
a designer 219 to create a synchronization script which includes annotation stream(s) . 
The annotation stream(s) define the content (s) of a LiveScreen display 245 to be 
displayed on client computer 240 for a viewer. LiveScreen display 245 provides a 
graphical user interface (GUI) with multiple windows for synchronously displaying a 
video stream from stream server 220 and at least one displayable event stream. Examples 
of displayable events include textual/graphical information such as HTML-scripted web 
page(s) from web server 23 0. 

Detailed Description Text (15) : 

In one embodiment, as shown in FIG. 3A, producer 215a includes a capture module 317a 
and an author module 318a. Production station 210 includes 16 MB of RAM and a 1 GB hard 
disk drive for capturing and storing an uncompressed or precompressed video stream. 
Sources for generating video streams include a video camera 312, a video cassette 
recorder (VCR) (not shown) or a previously digitized video file 314, e.g. , a Video for 
Windows (.avi) file. For ease of installation and use by designer 219, producer 215a is 
implemented in a host environment which includes a window-based operating system such 
as Microsoft Windows 95 and a web browser such as Netscape's Navigator 3.x. 

Detailed Description Text (16) : 

Referring also to the flowchart of FIG. 4A, in step 410 capture module 317a captures a 
live video/audio stream from video camera 312 or from the previously stored video file 
314. If video camera 312 provides an analog video stream, e.g., an NTSC signal, a 
hardware capture card (not shown) provides the required conversion from the analog 
video stream to a digitized video stream . Because temporary storage of uncompressed 
video data is memory intensive, some form of pre-compression can be used to reduce the 
memory storage requirement of the input video stream during capture step 410 and prior 
to compression step 420. 

Detailed Description Text (17) : 

In step 420, capture module 317a compresses the digitized video stream using a suitable 
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compression technique. In this embodiment, depending on the bandwidth capacity of the 
connection provided by network 2 90 between stream server 220 and client computer 240, 
e.g., a POTS modem, ISDN or Ethernet, a suitable frame resolution and frame rate 
combination is selected. FIG. 5 shows an exemplary format 500 for storing and 
delivering a compressed video stream . 

Detailed Description Text (21) : 

Designer 219 may view frames from video stream 500 displayed in video window 720 for 
referencing and selecting appropriate time stamps to use in generating annotation 
streams. Within video window 720, VCR function buttons, e.g., a rewind button 724, a 
play button 726 and a fast forward button 728, are available for designer 219 to 
quickly traverse video stream 500. Since video window 720 is provided as a convenience 
for designer 219, if designer 219 has prior knowledge of the content of the video 
stream, designer 219 may proceed with the generation of the annotation streams without 
viewing video window 720. 

Detailed Description Text (27) : 

In accordance with another aspect of the invention, LiveScreen display 600 also 
includes a table of contents (TOC) 630, enabling viewer 249 at client computer 240 to 
skip forward or backward to a point within the entire video/audio stream 500. TOC 630 
include one or more content labels, each indexed to a corresponding time stamp in video 
stream 500, as defined by TOC time markers 791, 792, 793, 794 in LiveScreen display 
600 . 



Detailed Description Text (28) : 

Referring now to FIG. 9A, in one embodiment of the present invention, client computer 
240 includes a web browser 950 and a browser plug-in module 952a for interfacing web 
browser 950 with a main client module 960. Client module 960 includes an event registry 
962, playout buffer(s) 966, video/audio decoder (s) 964, video /audio renderer(s) 965 and 
one or more dynamically loadable event applet(s), e.g., flipper applet 967, ticker 
applet 968 and VCR applet 969. In this embodiment, event registry 962 also functions as 
an annotation interpreter 963. 

Detailed Description Text (29) : 

FIG. 10A is a flowchart illustrating the operation of client module 960. Assume that 
viewer has not previously loaded client module 960 in client computer 240, but has 
already loaded a web browser 950, e.g., Netscape's Navigator (step 1010). Viewer surfs 
the world-wide web (www) via the internet and locates a web site of interest. 
Typically, the web site of interest is hosted on web server 230. Accordingly, a target 
web page is downloaded from web server 230 and displayed on client computer 240. 

Detailed Description Text (30) : 

The target web page includes a link to a customized LiveScreen display, e.g., display 
600. If client module 960 has not been previously loaded, client module 960 is now 
loaded over web browser 950 for processing video/audio and annotation streams (step 
1020) . Depending on the implementation, a copy of client module 960 may be available 
from the web site of interest. Alternatively, the target web page may provide a HTML 
link to another web server which has an updated copy of client module 960. 

Detailed Description Text (31) : 

Referring now to FIG. 10B, first, browser plug-in module 952a is installed over web 
browser 950 (step 1022) . As discussed above, plug-in module 952a provides the interface 
between client module 960 and web browser 950. The target web page provides a HTML link 
to the format for LiveScreen display 600. The LiveScreen display format is retrieved 
and display 600 is installed on client computer 240 using web browser 950 (step 1024) . 

Detailed Description Text (33) : 

Referring back to FIG. 10A, encoded video/audio frames and associated annotation frames 
are streamed from stream server 220 to client computer 240 for synchronous display 
(step 1030) . Streaming video and audio streams over a network is very efficient because 
streaming eliminates the need for a large buffer at client computer 240. In addition, 
streaming also provides flexibility, e.g., switching video sources midstream is 
possible without wasting network resources since streaming is based on a pseudo 
just-in-time (JIT) protocol and does not involve downloads of the entire video stream 
prior to display at client computer 240. If the underlying transmission protocol is 
HTTP, then video, audio and annotation packets are initially "pulled" by client 
computer 240 from server 220 using HTTP "get" packet (s). 

Detailed Description Text (34) : 
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Next, the encoded video/audio streams are decoded by decoder 964, i.e., decompressed 
using a suitable technique, and then displayed at client computer 240 by renderer 965 
(step 1040) . 

Detailed Description Text (37) : 

Further, since the video and annotation streams are handled synchronously but 
separately by video decoder 964 and annotation interpreter 963, respectively, steps 
1040 and 1050 can occur concurrently or consecutively. As discussed above, event 
registry 962 is capable of dynamic registration of event applets. Accordingly, 
annotation interpreter 963 is adaptable, and capable of automatic installation and 
linking of new event applet (s) to add new class (es) of displayable events for client 
computer 24 0 . 

Detailed Description Text (38) : 

After registering with event registry 962, flipper applet 967 provides the location of 
the flipper stream to broswer 950 which then begin receiving the flipper stream from 
stream server 220. Flipper annotation frames are provided by stream server 220 
synchronously with the video/audio frames to client module 960 so that the annotations, 
i.e., displayable events can be synchronized for display at client computer 240 (step 
1060) . In this example, URL addresses, for synchronizing HTML page flips with video 
stream are provided to web browser 950 thereby permitting client computer 240 to 
subsequently retrieve and display various textual and graphical elements changing at 
predetermined points corresponding to the timeline of the video stream . Note that HTML 
pages can be retrieved from one or more web server (s) 230. 

Detailed Description Text (39) : 

Similarly, after registering with event registry 962, ticker (tape) applet 968 provides 
the location of the ticker stream to broswer 950 which then begins receiving the ticker 
stream from stream server 220. Ticker annotation frames are provided by stream server 
220 synchronously with the video/audio frames so that the annotations i.e. 
displayable ticker data can be synchronized for display at client computer 240 at 
predetermined points corresponding to the timeline of the video stream . 

Detailed Description Text (42) : 

As shown in FIG. 11, a table of content 63 0 with content labels enables a viewer to 
skip forward or backward to predetermined locations in the video/audio stream. First, 
viewer selects a content label of interest (step 1110). Examples of suitable content 
labels are section headings of the video stream . Next, client module 960 sends a 
message to stream server 220 with the time stamp of an I -frame from the video stream 
whose location is close to selected content label (step 1120) . In this embodiment, an 
I -frame is a video frame which includes data for a complete video frame. Although 
computationally more intensive, it is also possible to select a P-frame and then 
reconstructed a complete video starting from a neighboring I -frame close to the 
selected P-frame. 

Detailed Description Text (44) : 

Referring now to FIGS. 3B and 9B, in another embodiment, instead of streaming three 
separate video, audio and annotation streams from stream server 220 to client computer 
240, an interleaved video/audio/annotation file is produced by producer 215b, stored in 
w eb server 230, and subsequently provided to client module 960 on demand via web 
browser 950. Note that an interleaved file can include any two or more frame types, 
e.g., video and audio frames, video and annotation frames, or audio and annotation 
frames . 

Detailed Description Text (45) : 

Advantages of this embodiment include simplified synchronous delivery of video, audio 
and annotation frames to client computer 240. Simplicity is accomplished by eliminating 
the need for stream server 220, whose primary function is to manage the transmission of 
several separate video, audio and annotation streams from stream server 220 to client 
computer 240. In this embodiment, since all the video, audio and annotation frames are 
combined into a single interleaved stream and are pre-sorted by timestamp values, the 
interleaved stream can now be stored in web server 230 and delivered in the form of 
HTTP data. 

Detailed Description Text (52) : 

In accordance with another aspect of this embodiment, the data packets 1320, 1330, 
. 1390 for streaming video and audio frames include a variable packet length field 1324 
as shown in FIGS. 13A and 13B. Referring to FIGS. 14A, 14B, and 14C, three exemplary 
formats 1324a, 1324b, and 1324c of the variable packet length field 1324 are shown In 
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this implementat ion, the length of the variable packet length field is in multiples of 
number units. For example, formats 1324a, 1324b and 1324c can be one numerical unit in 
length, three numerical units in length and seven numerical units in length, 
respectively. As is known to one skilled in the art, regardless of the size of the 
packet length field, the packet length can be represented by a number of different 
methods, such as simple binary, one's complement, BCD and floating point. 



CLAIMS : 



I . In a computer having a processor and memory, said computer useful in association 
with a web server coupled to a client computer via a network, a method for producing an 
interleaved multimedia file from a video file and an audio file, the method comprising: 
retrieving a first video frame from the video file, said first video frame including a 
video timestamp, in response to determining a video frame buffer is empty; retrieving a 
first audio frame from the audio file, said first audio frame including a audio 
timestamp, in response to determining an audio frame buffer is empty; writing the first 
video frame to a first packet of the interleaved file in response to determining that 
the video timestamp is less than or equal to the audio timestamp, retrieving a second 
video frame from the video file in response to determining that the video timestamp is 
less than or equal to the audio timestamp, writing the first audio frame to a second 
packet of the interleaved file in response to determining that the audio timestamp is 
less than or equal to the video timestamp, and retrieving a second audio frame from the 
audio file in response to determining that the audio timestamp is less than or equal to 
the video timestamp; writing a size of a variable packet length field of the first 
packet of the interleaved multimedia file to a length field of the variable packet 
length field of the first packet of the interleaved multimedia file; and writing at 
least one null number into a null field of the variable packet length field of the 
first packet of the interleaved multimedia file, a number of null numbers written into 
the null field being selected as a function of a size of the first video frame. 

3. In a computer having a processor and memory, said computer useful in association 
with a web server coupled to a client computer via a network, a method for producing an 
interleaved multimedia file from a video file and an audio file, the method comprising: 
retrieving a first video frame from the video file, said first video frame including a 
video timestamp, upon determining that a video frame buffer is empty; retrieving a 
first audio frame from the audio file, said first audio frame including an audio 
timestamp, upon determining that an audio frame buffer is empty; and writing the first 
video frame to a first packet of the interleaved file upon determining that the video 
timestamp is less than or equal to the audio timestamp, retrieving a second video frame 
from the video file upon determining that the video timestamp is less than or equal to 
the audio timestamp, writing the first audio frame to a second packet of the 
interleaved file upon determining that the audio timestamp is less than or equal to the 
video timestamp, and retrieving a second audio frame from the audio file upon 
determining that the audio timestamp is less than or equal to the video timestamp, 
wherein said first and second packet each have a variable packet length field having a 
size, wherein the size of the variable packet length field is at least one numerical 
unit, and writing the first video frame to the first packet includes writing the size 
into a length of the variable packet length field of the first packet, writing a null 
number into a null field of the variable packet length field of the first packet upon 
determining the size of the first video frame is between one numerical unit and two 
numerical units, and writing three null numbers into the null field of the variable 
packet length field of the first packet upon determining the size of the first video 
frame is greater than two numerical units. 

II. In a computer having a processor and memory, said computer useful in association 
with a web server coupled to a client computer via a network, a method for producing an 
interleaved multimedia file from a video file and an audio file, the method comprising: 
retrieving a first video frame from the video file, said first video frame including a 
video timestamp, upon determining that a video frame buffer is empty; retrieving a 
first audio frame from the audio file, said first audio frame including an audio 
timestamp, upon determining that an audio frame buffer is empty; and writing the first 
video frame to a first packet of the interleaved file upon determining that the video 
timestamp is less than or equal to the audio timestamp; retrieving a second video frame 
from the video file upon determining that the video timestamp is less than or equal to 
the audio timestamp; writing the first audio frame to a second packet of the 
interleaved file upon determining that the audio timestamp is less than or equal to the 
video timestamp; and retrieving a second audio frame from the audio file upon 
determining that the audio timestamp is less than or equal to the video timestamp, 
wherein said first and second packet each have a variable packet length field having a 
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size, wherein the size of the variable packet length field is at least one numerical 
unit, and writing the first video frame to the first packet includes writing the size 
into a length field of the variable packet length field of the first packet, and 
writing a number into a field of the variable packet length field, wherein the number 
corresponds to the size of the video frame. 

13 . In a computer having a processor and a memory, the computer useful in association 
with a web server coupled to a client computer via a network, a method for producing an 
interleaved multimedia file from a video file and an audio file, the method comprising: 
retrieving a first video frame from the video file in response to determining that a 
video frame buffer is empty, the first video frame having a video timestamp; retrieving 
a first audio frame from the audio file in response to determining that an audio frame 
buffer is empty, the first audio frame having an audio timestamp; writing the first 
video frame to a first packet of the interleaved multimedia file in response to 
determining that the video timestamp is not greater than the audio timestamp; 
retrieving a second video frame from the video file in response to determining that the 
video timestamp is not greater than the audio timestamp; writing the first audio frame 
to a second packet of the interleaved multimedia file in response to determining that 
the audio timestamp is not greater than the video timestamp; retrieving a second audio 
frame from the audio file in response to determining that the audio timestamp is not 
greater than the video timestamp; and representing a length of a variable packet length 
field of a packet of the interleaved multimedia rule such that the length can be 
determined from one or more null numbers included within the variable packet length 
field. 

14 . In a computer having a processor and a memory, the computer useful in association 
with a web server coupled to a client computer via a network, a method for producing an 
interleaved multimedia file from a video file and an audio file, the method comprising: 
retrieving a first video frame from a video file in response to determining that a 
video frame buffer is empty, the first video frame having a video timestamp; retrieving 
a first audio frame from an audio file in response to determining that an audio frame 
buffer is empty, the first audio frame having an audio timestamp; writing the first 
video frame to a first packet of the interleaved multimedia file in response to 
determining that the video timestamp is not greater than the audio timestamp; 
retrieving a second video frame from the video file in response to determining that the 
video timestamp is not greater than the audio timestamp; writing the first audio frame 
to a second packet of the interleaved multimedia file in response to determining that 
the audio timestamp is not greater than the video timestamp; retrieving a second audio 
frame from the audio file in response to determining that the audio timestamp is not 
greater than the video timestamp; writing a size of a variable packet length field of 
the first packet of the interleaved multimedia file to a length field of the variable 
packet length field of the first packet of the interleaved multimedia file; and writing 
at least one reserved value to the variable packet length field of the first packet of 
the interleaved multimedia file, a number of reserved values written being selected as 
a function of a size of the first video frame. 
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DOCUMENT- IDENTIFIER: US 6006241 A 

TITLE: Production of a video stream with synchronized annotations over a computer 
network 



Abstract Text (1) : 

The production of synchronization scripts and associated annotated multimedia streams 
for servers and client computers coupled to each other by a diverse computer network 
which includes local area networks (LANs) and/or wide area networks (WANs) such as the 
internet. Annotated multimedia streams can include a compressed video stream for 
display in a video window, an accompanying compressed audio stream and annotations. 
Synchronization scripts include annotation streams for synchronizing the display of 
video streams with annotations, e.g., displayable events, such textual/graphical data 
in the form of HTML pages with Java applets to be displayed in one or more event 
windows. The producer includes a capture module and an author module for capturing 
video streams and generating annotation streams, respectively. The capture module 
compresses the video stream using a suitable compression format. Annotation streams 
include annotation frames which provide either pointer (s) to the event (s) of interest 
or include displayable data embedded within the annotation stream. Accordingly, each 
annotation frame includes either an event locator or an event data. In addition, each 
annotation frame includes an event time marker which corresponds to the time stamp (s) 
of associated video frame (s) within the video stream . Embedded displayable data include 
ticker tape data embedded within the annotation stream. Examples of event locators to 
displayable events include URL addresses pointing to HTML web pages. The video/audio 
streams and annotation streams are stored in stream server (s) for subsequent retrieval 
by client computer (s) in a coordinated manner, so that the client computer (s) is able 
to synchronously display the video frames and displayable event (s) in a video window 
and event window ( s) , respectively. In one implementation, annotation streams include a 
flipper stream for locating HTML pages and a ticker stream which include ticker (tape) 
data . 

Application Filing Date (1) : 
19970314 

Parent Case Text (1) : 

This application is related to co-pending U.S. application Ser. No. 08/818,805, filed 
on Mar. 14, 1997, entitled "Method and Apparatus for Implementing Motion Detection in 
Video Compression", non-final action mailed May 12, 1999 U.S. application Ser. No. 
08/819/507, filed on Mar. 14, 1997, entitled "Digital Video Signal Encoder and Encoding 
Method", final rejection mailed Jun. 3, 1999 U.S. application Ser. No. 08/818,804, 
filed on Mar. 14, 1997, entitled "Production of a Video Stream with Synchronized 
Annotations over a Computer Network", case allowed Jun. 7, 1999 U.S. application Ser. 
No. 08/819,586, filed on Mar. 14, 1997, entitled "Method and Apparatus for Implementing 
Control Functions in a Streamed Video Display System", Response to non-final Enter May 
6, 1999 U.S.' application Ser. No. 08/818,769, filed on Mar. 14, 1997, entitled "Method 
and Apparatus for Automatically Detecting Protocols in a Computer Network", Response to 
non-final entered U.S. application Ser. No. 08/818,127, filed on Mar. 14, 1997, 
entitled "Dynamic Bandwidth Selection for Efficient Transmission of Multimedia Streams 
in a Computer Network", Final rejection mailed Apr. 27, 1999 U.S. application Ser. No. 
08/819,585, filed on Mar. 14, 1997, entitled "Streaming and Display of a Video Stream 
with Synchronized Annotations over a Computer Network" , case available for examination 
U.S. application Ser. No. 08/818,664, filed on Mar. 14, 1997, entitled "Selective 
Retransmission for Efficient and Reliable Streaming of Multimedia Packets in a Computer 
Network", Allowed Jun. 1, 1999 U.S. application Ser. No. 08/819,579, filed on Mar. 14, 
1997, entitled "Method and Apparatus for Table-Based Compression with Embedded Coding", 
Incomplete application- -mailed Jun. 16, 1999 U.S. application Ser. No. 08/819,587, 
filed on Mar. 14, 1997, entitled "Method and Apparatus for Implementing Motion 
Estimation in Video Compression", Appeal Brief filed May 5, 1999 U.S. application Ser. 
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No. 08/818,826, filed on Mar. 14, 1997, entitled "Digital Video Signal Encoder and 
Encoding Method", all filed concurrently herewith, patented file May 1, 1999 U.S. 
application Ser. No. 08/822,156, filed on Mar. 17, 1997, entitled "Method and Apparatus 
for Communication Media Commands and Data Using the HTTP Protocol", Response to 
non-final office Action Mar. 25, 1999 provisional U.S. Application Ser. No. 60/036,662, 
filed on Jan. 30, 1997, entitled "Methods and Apparatus for Autodetecting Protocols in 
a Computer Network" application Expired Mar. 20, 1999 U.S. application Ser. No. 
08/625,650 filed on Mar. 29, 1996, entitled "Table-Based Low-Level Image Classification 
System", Response to non-final office Action entered Jun. 9, 1999 U.S. application Ser. 
No. 08/714,447, filed on Sep. 16, 1996, entitled "Multimedia Compression System with 
Additive Temporal Layers", and is a continuation-in-part of U.S. application Ser. No. 
08/623,299, non-final Action mailed Feb. 19, 1999 filed on Mar. 28, 1996, entitled 
"Table-Based Compression with Embedded Coding", which are all incorporated by reference 
in their entirety for all purposes incomplete application- -mailed Jun. 10, 1999. 

Brief Summary Text (6) : 

Existing conventional internet applications, such as electronic mailers and web 
browser, are capable of transferring and presenting textual and graphical information. 
However, none of these individual internet applications effectively provide synchronous 
delivery of a combination of diverse multimedia streams in a coherent and integrated 
manner. This is because executing several independent and unrelated applications to 
present the diverse combination of multimedia streams on a client computer can result 
in a hodgepodge of poor quality, incompatible and/or incoherent presentations. 

Brief Summary Text (7) : 

In view of the foregoing, there are desired improved techniques for reliably providing 
a multimedia stream such as a video and audio stream, together with annotations such as 
textual and graphical information in an integrated seamless package to client 
computer(s), while efficiently utilizing the network resources and consuming minimal 
computational cycles on the client computer (s) . 

Brief Summary Text (9) : 

The present invention provides synchronization scripts and associated annotated 
multimedia streams for servers and client computers coupled to each other by a diverse 
computer network which includes local area networks (LANs) and/or wide area networks 
(WANs) such as the internet. Annotated multimedia streams can include a compressed 
video stream for display in a video window, an accompanying compressed audio stream and 
annotations. Synchronization scripts include annotation streams for synchronizing the 
display of video streams with annotations, e.g., displayable events, such 
textual/graphical data in the form of HTML pages with Java applets to be displayed in 
one or more event windows . 



Brief Summary Text (10) : 

In one embodiment, a producer includes a capture module and an author module for 
capturing video streams and generating annotation streams, respectively. The video and 
annotation streams are then stored in stream server (s) to be provided to one or more 
client computer (s) upon request. 

Brief Summary Text (11) : 

The capture module compresses the video stream using a compression format based on a 
standard H263, generating, for example, a QCIF resolution ( 176 . times . 144) video frames 
at 10-20 frames per second (fps) which can be encoded and transmitted over a 20 Kbps 
connection. Alternatively, using the scalable vector quantization (SVQ) compression 
algorithm of the present invention, dynamically scalable data transmission rates from 
10 Kbps to several Mbps can support scalable resolutions of 160 . times . 120 to 
640 . times . 480 and frames rates ranging from 1 fps to 30 fps. Other compression 
techniques can also be used with the present invention. 

Brief Summary Text (12) : 

In this embodiment, annotation streams include annotation frames which provide either 
pointer (s) to the event (s) of interest or include displayable data embedded within the 
annotation stream. Accordingly, each annotation frame includes either an event locator 
or an event data. In addition, each annotation frame includes an event time marker 
which corresponds to the time stamp (s) of associated video frame (s) within the video 
stream . Examples of embedded displayable data include ticker tape data embedded within 
the annotation stream. Examples of event locators to displayable events include URL 
addresses pointing to HTML web pages. Note that an event time marker need not be 
identical to a corresponding video time stamp. The client computer is capable of 
switching to a new displayable event together with a video frame or in between two 
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video frames. 

Drawing Description Text (6) : 

FIG. 4A is a flowchart illustrating the capture of a live video/audio stream from a 
video camera or from a previously stored video file. 

Drawing Description Text (8) : 

FIG. 5 shows an exemplary format for storing and delivering a compressed video stream . 
Drawing Description Text (12) : 

FIG. 9 illustrates one embodiment of the client computer which includes a web browser 
and a browser plug- in module for interfacing a web browser with a client module. 

Detailed Description Text (13) : 

FIG. 2 is a block diagram showing an exemplary hardware environment for practicing the 
annotated video -on-demand (VOD) system of the present invention. The VOD system 
includes a production station 210 , a stream server 220, at least one web server 230 and 
at least one client computer 240, each of which can be implemented using computer 
system 100 described above. Stream server 220 and web server 230 are coupled to client 
computer 240 via a computer network 2 90, e.g., the internet. Note that the disclosed 
hardware environment is exemplary. For example, production station 210 and stream 
server 22 0 can be implemented using two separate computer systems or using one computer 
system. In addition, if production station 210 and stream server 220 are implemented on 
separate computer systems as shown in FIG. 2, an optional direct connection (not shown) 
between production station 210 and stream server 220 can provide faster uploads of 
compressed video and annotation streams. In the following description, an audio stream 
optionally accompanies each video stream . 

Detailed Description Text (14) : 

A producer 215, installed in production station 210, is a user-friendly tool for use by 
a designer 219 to create a synchronization script which includes annotation stream (s) . 
The annotation stream(s) define the content (s) of a LiveScreen display 245 to be 
displayed on client computer 240 for a viewer 249. LiveScreen 245 display provides a 
graphical user interface (GUI) with multiple windows for synchronously displaying a 
video stream from stream server 220 and at least one displayable event stream. Examples 
of displayable events include textual/graphical information such as HTML- scripted web 
page(s) from web server 230. 

Detailed Description Text (15) : 

In one embodiment, as shown in FIG. 3, producer 215 includes a capture module 317 and 
an author module 318. Production station 210 includes 16 MB of RAM and a 1 GB hard disk 
drive for capturing and storing an uncompressed or precompressed video stream . Sources 
for generating video streams include a video camera 312, a video cassette recorder 
(VCR) (not shown) or a previously digitized video file 314, e.g., a Video for Windows 
(.avi) file. For ease of installation and use by designer 219, producer 215 is 
implemented in a host environment which includes a window-based operating system such 
as Microsoft Windows 95 and a web browser such as Netscape's Navigator 3.x. (Appendix A 
is a detailed user manual for one implementation of producer 215) . 

Detailed Description Text (16) : 

Referring also to the flowchart of FIG. 4A, in step 410 capture module 317 captures a 
live video/audio stream from video camera 312 or from the previously stored video file 
314. If video camera 312 provides an analog video stream, e.g., an NTSC signal, a 
hardware capture card (not shown) provides the required conversion from the analog 
video stream to a digitized video stream . Because temporary storage of uncompressed 
video data is memory intensive, some form of pre-compression can be used to reduce the 
memory storage requirement of the input video stream during capture step 410 and prior 
to compression step 420. 

Detailed Description Text (17) : 

In step 420, capture module 420 compresses the digitized video stream using a suitable 
compression technique. In this embodiment, depending on the bandwidth capacity of the 
connection provided by network 290 between stream server 220 and client computer 240, 
e.g., a POTS modem, ISDN or Ethernet, a suitable frame resolution and frame rate 
combination is selected. A compression algorithm based on the H263 standard (see 
co-pending applications VXT 702 and 718) is used for compressing lower bandwidth video 
streams, e.g., at less than 56 kbps . Alternatively, a Vxpress format (see co-pending 
application VXT 712) is used for compressing higher bandwidth video streams . FIG. 5 
shows an exemplary format 500 for storing and delivering a compressed video stream. 
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Detailed Description Text (21) : 

Designer 219 may view frames from video stream 500 displayed in video window 720 for 
referencing and selecting appropriate time stamps to use in generating annotation 
streams. Within video window 720, VCR function buttons, e.g., a rewind button 724, a 
play button 726 and a fast forward button 728, are available for designer 219 to 
quickly traverse video stream 500. Since video window 720 is provided as a convenience 
for designer 219, if designer 219 has prior knowledge of the content of the video 
stream, designer 219 may proceed with the generation of the annotation streams without 
viewing video window 720. 

Detailed Description Text (27) : 

In accordance with another aspect of the invention, LiveScreen display 600 also 
includes a table of contents (TOC) 630, enabling viewer 249 at client computer 240 to 
skip forward or backward to a point within the entire video/audio stream 500. TOC 630 
include one or more content labels, each indexed to a corresponding time stamp in video 
stream 500, as defined by TOC time markers 791, 792, 793, 794 in LiveScreen display 
600. 

Detailed Description Text (28) : 

Referring now to FIG. 9, in one embodiment of the present invention, client computer 
240 includes a web browser 950 and a browser plug-in module 952 for interfacing web 
browser 950 with a main client module 960. Client module 960 includes an event registry 
962, playout buffer (s) 966, video/audio decoder (s) 964, video /audio renderer(s) 965 and 
one or more dynamically loadable event applet(s), e.g., flipper applet 967, ticker 
applet 968 and VCR applet 969. In this embodiment, event registry 962 also functions as 
an annotation interpreter 963 . 

Detailed Description Text (29) : 

FIGS. 10A is a flowchart illustrating the operation of client module 960. Assume that 
viewer 249 has not previously loaded client module 960 in client computer 240, but has 
already loaded a web browser 950, e.g., Netscape's Navigator (step 1010). Viewer 249 
surfs the world-wide web (www) via the internet and locates a web site of interest to 
viewer 249. Typically, the web site of interest is hosted on web server 230. 
Accordingly, a target web page is downloaded from web server 23 0 and displayed on 
client computer 240. 

Detailed Description Text (30) : 

The target web page includes a link to a customized LiveScreen display, e.g., display 
600. If client module 960 has not been previously loaded, client module 960 is now 
loaded over web browser 950 for processing video/audio and annotation streams (step 
1020) . Depending on the implementation, a copy of client module 960 may be available 
from the web site of interest. Alternatively, the target web page may provide a HTML 
link to another web server which has an updated copy of client module 960. 

Detailed Description Text (31) : 

Referring now to FIG. 10B, first, browser plug-in module 952 is installed over web 
browser 950 (step 1022) . As discussed above, plug-in module 952 provides the interface 
between client module 960 and web browser 950. The target web page provides a HTML link 
to the format for LiveScreen display 600. The LiveScreen display format is retrieved 
and display 600 is installed on client computer 240 using web browser 950 (step 1024) . 

Detailed Description Text (33) : 

Referring back to FIG. 10A, encoded video/audio frames and associated annotation frames 
are streamed from stream server 220 to client computer 240 for synchronous display 
(step 1030) . Streaming video and audio streams over a network is very efficient because 
streaming eliminates the need for a large buffer at client computer 240. In addition, 
streaming also provides flexibility, e.g., switching video sources midstream is 
possible without wasting network resources since streaming is based on a pseudo 
just-in-time (JIT) protocol and does not involve downloads of the entire video stream 
prior to display at client computer 240. If the underlying transmission protocol is 
HTTP, then video, audio and annotation packets are initially "pulled" by client 
computer 240 from server 220 using HTTP "get" packet (s). 

Detailed Description Text (34) : 

Next, the encoded video/audio streams are decoded by decoder 964, i.e., decompressed 
using a suitable technique, and then displayed at client computer 240 by renderer 965 
(step 1040) . (See co-pending applications VXT 702, 712 and 718) . 
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Detailed Description Text (37) : 

Further, since the video and annotation streams are handled synchronously but 
separately by video decoder 964 and annotation interpreter 963, respectively, steps 
1040 and 1050 can occur concurrently or consecutively. As discussed above, event 
registry 962 is capable of dynamic registration of event applets. Accordingly, 
annotation interpreter 963 is adaptable, and capable of automatic installation and 
linking of new event applet (s) to add new class (es) of displayable events for client 
computer 240. 

Detailed Description Text (38) : 

After registering with event registry 962, flipper applet 967 provides the location of 
the flipper stream to browser 950 which then begin receiving the flipper stream from 
stream server 220. Flipper annotation frames are provided by stream server 220 
synchronously with the video/audio frames to client module 960 so that the annotations, 
i.e., displayable events can be synchronized for display at client computer 240 (step 
1060) . In this example, URL addresses, for synchronizing HTML page flips with video 
stream are provided to web browser 950 thereby permitting client computer 240 to 
subsequently retrieve and display various textual and graphical elements changing at 
predetermined points corresponding to the timeline of the video stream . Note that HTML 
pages can be retrieved from one or more web server (s) 230. 

Detailed Description Text (39) : 

Similarly, after registering with event registry 962, ticker (tape) applet 968 provides 
the location of the ticker stream to browser 950 which then begins receiving the ticker 
stream from stream server 220. Ticker annotation frames are provided by stream server 
220 synchronously with the video/audio frames so that the annotations, i.e., 
displayable ticker data can be synchronized for display at client computer 240 at 
predetermined points corresponding to the timeline of the video stream . 

Detailed Description Text (42) : 

As shown in FIG. 11, a table of content 630 with content labels enables viewer 249 to 
skip forward or backward to predetermined locations in the video/audio stream. First, 
viewer 249 selects a content label of interest (step 1110) . Examples of suitable 
content labels are section headings of the video stream . Next, client module 960 sends 
a message to stream server 220 with the time stamp of an I-frame from the video stream 
whose location is close to selected content label (step 1120) . In this embodiment, an 
I-frame is a video frame which includes data for a complete video frame. Although 
computationally more intensive, it is also possible to select a P-frame and then 
reconstructed a complete video starting from a neighboring I-frame close to the 
selected P-frame. 

CLAIMS : 

1. In a computer system, a method for generating a synchronization script configured to 
orchestrate a synchronized display of a video stream and displayable elements on a 
display device of a client computer, said video stream including a plurality of video 
frames, each said video frame associated with a time stamp, the method comprising: 

generating a first annotation stream for inclusion in said synchronization script, said 
first annotation stream including a first plurality of annotation frames, said first 
plurality of annotation frames associated with a first selected subset of said video 
frames, each said first plurality of annotation frames including an event locator and 
an event time marker, each said event locator pointing to the location of one of a 
first plurality of displayable elements stored in a web server coupled to said client 
computer, said event time markers of said first plurality of annotation frames 
corresponding to the respective time stamps of said first selected subset of video 
frames . 

5. In a computer system, a method for generating a synchronization script configured to 
orchestrate a synchronized display of a video stream and displayable elements on a 
display device of a client computer, said video stream including a plurality of video 
frames, each said video frame associated with a time stamp, the method comprising: 

generating at least one annotation stream for inclusion in said synchronization script, 
said at least one annotation stream including a plurality of annotation frames, said 
annotation frames associated with a selected subset of said video frames, each said 
annotation frame including a corresponding one of said displayable events and an event 
time marker, said time markers of said annotation frames corresponding to the 
respective time stamps of said selected subset of video frames. 
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7. In a client computer having a display device, a method of skipping forward or 
backward to a predetermined one of a plurality of video frames of a video stream being 
displayed on said display device, each said video frame including a time stamp, and 
wherein a table of contents is also displayed on said display device, the method 
comprising : 

receiving said video stream from a stream server coupled to the client computer via a 
computer network; 

sequentially displaying said video frames on said display device, starting from an 
initial video frame of said video stream ; 

selecting a content label from said table of contents, said content label providing an 
index into an intermediate video frame of said video stream, said index based on the 
time stamp of said intermediate video frame; 

communicating said index to said stream server; 

receiving said video stream from said stream server, starting from said intermediate 
video frame ; and 

sequentially displaying said video frames on said display device, starting with said 
intermediate video frame. 

8. In a stream server, a method of causing a client computer to skip forward or 
backward to a predetermined one of a plurality of video frames of a video stream being 
displayed on a display device of said client computer, said stream server coupled to 
the client computer via a computer network, each said video frame including a time 
stamp, and wherein a table of content is also displayed on said display device, the 
method comprising: 

streaming said video stream to said client computer for sequential display on said 
display device, starting from an initial video frame of said video stream ; 

receiving an index from said client computer, said index associated with a selectable 
content label from a table of contents displayed on said display device, said index 
based on the time stamp of an intermediate video frame of said video stream ; and 

streaming said video stream to said client computer for sequential display on said 
display device, starting from said intermediate video frame. 

9. A producer useful for generating a synchronization script configured to orchestrate 
a synchronized display of a video stream and displayable elements on a display device 
of a client computer, the producer comprising: 

a capture module configured to capture said video stream which includes a plurality of 
video frames, each said video frame associated with a time stamp; and 

an author module configured to generate a first annotation stream for inclusion in said 
synchronization script, said first annotation stream including a first plurality of 
annotation frames, said first plurality of annotation frames associated with a first 
selected subset of said video frames, each said first plurality of annotation frames 
including an event locator and an event time marker, each said event locator pointing 
to the location of one of a first plurality of displayable elements stored in a web 
server coupled to said client computer, said event time markers of said first plurality 
of annotation frames corresponding to the respective time stamps of said first selected 
subset of video frames. 

13. A producer useful for generating a synchronization script configured to orchestrate 
a synchronized display of a video stream and displayable elements on a display device 
of a client computer, the producer comprising: 

a capture module configured to capture said video stream which includes a plurality of 
video frames, each said video frame associated with a time stamp; and 

an author module configured to generate at least one annotation stream for inclusion in 
said synchronization script, said at least one annotation stream including a plurality 
of annotation frames, said annotation frames associated with a selected subset of said 
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video frames, each said annotation frame including a corresponding one of said 
displayable events and an event time marker, said time markers of said annotation 
frames corresponding to the respective time stamps of said selected subset of video 
frames . 

15. A client computer useful in association with a stream server coupled to the client 
computer via a computer network, the client computer comprising: 

a playout buffer configured to receive a video stream from said stream server, said 
video stream including a plurality of video frames, each said video frame including a 
time stamp; 

a display device configured to sequentially display said video frames on said display 
device, starting from an initial video frame of said video stream, said display device 
also configured to display a selectable content label from said table of contents, said 
selectable content label providing an index into an intermediate video frame of said 
video stream, said index based on the time stamp of said intermediate video frame; and 

a transmitter configured to communicate said index to said stream server, thereby 
causing said stream server to begin transmitting said video stream to said client 
computer, starting from said intermediate video frame, and causing said display device 
to sequentially display said video frames, starting with said intermediate video frame. 



16. A stream server useful in association with a client computer coupled to said stream 
server via a computer network, the stream server comprising: 

a streamer configured to store and stream a video stream to said client computer, said 
video stream including a plurality of video frames, said video stream enabling said 
client computer to sequentially display said video frames on a display device of said 
client computer, starting from an initial video frame of said video stream ; and 

a receiver configured to receive an index from said client computer, said index 
associated with a selectable content label from a table of contents displayed on said 
display device, said index based on the time stamp of an intermediate video frame of 
said video stream, and wherein said received index causes said streamer to begin 
streaming said video stream to said client computer, and enabling said client computer 
to sequentially display said video frames on said display device starting from said 
intermediate video frame. 

17. A computer -readable medium useful in association with a computer system, the 
computer-readable medium having computer- executable instructions which cause the 
computer system to perform a method comprising generating a first annotation stream 
which includes a first plurality of annotation frames, said first plurality of 
annotation frames associated with a first selected subset of a plurality of video 
frames of a video stream, each said video frame associated with a time stamp, each said 
first plurality of annotation frames including an event locator and an event time 
marker, each said event locator pointing to the location of one of a first plurality of 
displayable elements stored in a web server coupled to said client computer, said event 
time markers of said first plurality of annotation frames corresponding to the 
respective time stamps of said first selected subset of video frames. 

21. A computer-readable medium useful in association with a computer system, the 
computer -readable medium having computer-executable instructions which cause the 
computer system to perform a method comprising generating at least one annotation 
stream which includes a plurality of annotation frames, said annotation frames 
associated with a selected subset of a plurality of video frames of a video stream, 
each said video frame associated with a time stamp, each said annotation frame 
including a corresponding one of said displayable events and an event time marker, said 
time markers of said annotation frames corresponding to the respective time stamps of 
said selected subset of video frames. 



7 of 7 



9/12/03 9:40 PM 



Record Display Form 




http://westbrs:8002^n/gate.exe?f=TOC8. 




;PAB,DWPIJDBD&ESNAME=KWIC&TOTAL REC=26 




Generate Collection 



L10 : Entry 14 of 26 



File: USPT 



Jan 9, 2001 



DOCUMENT- IDENTIFIER: US 6173317 Bl 

TITLE: Streaming and displaying a video stream with synchronized annotations over a 
computer network 

Abstract Text (1) : 

Client computer (s) retrieve and display synchronized annotated multimedia streams from 
servers dispersed over a diverse computer network: which includes local area networks 
(LANs) and/or wide area networks (WANs) such as the internet. Multimedia streams 
provided to the client computer (s) can include a compressed video stream for display in 
a video window and an accompanying compressed audio stream. Annotations, i.e., 
displayable events, include textual/graphical data in the form of HTML pages with Java 
applets to be displayed in one or more event windows. The video/audio and annotation 
streams are produced and then stored in stream server (s) . Annotation streams include 
annotation frames which provide either pointer (s) to the event (s) of interest or 
include displayable data embedded within the annotation stream. Accordingly, each 
annotation frame includes either an event locator or an event data. In addition, each 
annotation frame includes an event time marker which corresponds to the time stamp (s) 
of associated video frame (s) within the video stream . Examples of embedded displayable 
data include ticker tape data embedded within the annotation stream. Examples of event 
locators to displayable events include URL addresses pointing to HTML web pages. 
Video/audio streams and annotation streams are provided by the stream server (s) to the 
client computer (s) in a coordinated manner, so that the client computer (s) is able to 
synchronously display the video frames and displayable event (s) in a video window and 
event window(s), respectively. 

Application Filing Date (1) : 
19970314 

Parent Case Text (2) : 

This application is related to co-pending U.S. application Ser. No. 08/818,805, filed 
on Mar. 14, 1997, entitled "Method and Apparatus for Implementing Motion Detection in 
Video Compression," U.S. application Ser. No. 08/819,507, filed Mar. 14, 1997, entitled 
"Digital Video Signal Encoder and Encoding Method, " U.S. application Ser. No. 
08/818,804, filed on Mar. 14, 1997, entitled "Production of a Video Stream with 
Synchronized Annotations over a Computer Network," U.S. application Ser. No. 
08/819,586, filed on Mar. 14, 1997, entitled "Method and Apparatus for Implementing 
Control Functions in a Streamed Video Display System," U.S. application Ser. No. 
08/818,769, filed on Mar. 14, 1997, entitled "Method and Apparatus for Automatically 
Detecting Protocols in a Computer Network," U.S. application Ser. No. 08/818,127, filed 
on Mar. 14, 1997, entitled "Dynamic Bandwidth Selection for Efficient Transmission of 
Multimedia Streams in a Computer Network," U.S. application Ser. No. 08/819,585, filed 
on Mar. 14, 1997, entitled "Streaming and Display of a Video Stream with Synchronized 
Annotations over a Computer Network," U.S. application Ser. No. 08/818,664, filed on 
Mar. 14, 1997, entitled "Selective Retransmission for Efficient and Reliable Streaming 
of Multimedia Packets in a Computer Network," U.S application Ser. No. 08/819,579, 
filed Mar. 14, 1997, entitled "Method and Apparatus for Table-Based Compression with 
Embedded Coding," U.S. application Ser. No. 08/818,826, filed on Mar. 14, 1997, 
entitled "Digital Video Signal Encoder and Encoding Method," all filed concurrently 
herewith, U.S. application Ser. No. 08/822,156, filed on Mar. 17, 1997, entitled 
"Method and Apparatus for Communication Media Commands and Data Using the HTTP 
Protocol," provisional U.S. application Ser. No. 60/036,662, filed on Jan. 30, 1997, 
entitled "Methods and Apparatus for Autodetecting Protocols in a Computer Network, " 
U.S. application Ser. No. 08/625,650, filed on Mar. 29, 1996, entitled "Table-Based 
Low-Level Image Classification System," U.S. application Ser. No. 08/714,447, filed on 
Sep. 16, 1996, entitled "Multimedia Compression System with Additive Temporal Layers, 11 
and is a continuation-in-part of U.S. application Ser. No. 08/623,299, filed on Mar. 
28, 1996, entitled "Table-Based Compression with Embedded Coding," which are all 
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incorporated by reference in their entirety for all purposes. 
Brief Summary Text (6) : 

Existing conventional internet applications, such as electronic mailers and web 
browers, are capable of transferring and presenting textual and graphical information. 
However, none of these individual internet applications effectively provide synchronous 
delivery of a combination of diverse multimedia streams in a coherent and integrated 
manner. This is because executing several independent and unrelated applications to 
present the diverse combination of multimedia streams on a client computer can result 
in a hodgepodge of poor quality, incompatible and/or incoherent presentations. 

Brief Summary Text (7) : 

In view of the foregoing, there are desired improved techniques for reliably providing 
a multimedia stream such as a video and audio stream, together with annotations such as 
textual and graphical information in an integrated seamless package to client 
computer (s), while efficiently utilizing the network resources and consuming minimal 
computational cycles on the client computer (s) . 

Brief Summary Text (9) : 

The present invention enables client computer (s) to retrieval and display synchronized 
annotated multimedia streams from servers dispersed over a diverse computer network 
which includes local area networks (LANs) and/or wide area networks (WANs) such as the 
internet. Multimedia streams provided to the client computer (s) can include a 
compressed video stream for display in a video window and an accompanying compressed 
audio stream. Annotations, i.e., displayable events, include textual/graphical data in 
the form of HTML pages with Java applets to be displayed in one or more event windows. 

Brief Summary Text (10) : 

In one embodiment, the video/audio and annotation streams are produced by a capture 
module and an author module, and then stored in stream server (s) to be provided to one 
or more client computer (s) upon request. The capture module compresses the video stream 
using a suitable compression format, depending on the desired resolution (s) and frame 
rate(s) . The author module then generates synchronization scripts which include 
annotation streams which are synchronized with the compressed video/audio streams. 

Brief Summary Text (11) : 

In this embodiment, annotation streams include annotation frames which provide either 
pointer (s) to the event (s) of interest or include displayable data embedded within the 
annotation stream. Accordingly, each annotation frame includes either an event locator 
or an event data. In addition, each annotation frame includes an event time marker 
which corresponds to the time stamp (s) of associated video frame (s) within the video 
stream . Examples of embedded displayable data include ticker tape data embedded within 
the annotation stream. Examples of event locators to displayable events include URL 
addresses pointing to HTML web pages. Note that an event time marker need not be 
identical to a corresponding video time stamp. The client computer is capable of 
switching to a new displayable event together with a video frame or in between two 
video frames . 

Brief Summary Text (13) : 

The client computer also provides a table of contents displayed concurrently with the 
video window and the event window. The table of contents includes content labels which 
enable the viewer to skip forward or backward to one or more predetermined locations in 
the video stream . 

Drawing Description Text (5) : 

FIG. 4A is a flowchart illustrating the capture of a live video/audio stream from a 
video camera or from a previously stored video file. 

Drawing Description Text (7) : 

FIG. 5 shows an exemplary format for storing and delivering a compressed video stream . 
Drawing Description Text (11) : 

FIG. 9 illustrates one embodiment of the client computer which includes a web browser 
and a browser plug- in module for interfacing a web browser with a client module. 

Detailed Description Text (13) : 

FIG. 2 is a block diagram showing an exemplary hardware environment for practicing the 
annotated video -on-demand (VOD) system of the present invention. The VOD system 
includes a production station 210, a stream server 220, at least one web server 230 and 
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at least one client computer 240, each of which can be implemented using computer 
system 100 described above. Stream server 220 and web server 23 0 are coupled to client 
computer 240 via a computer network 290, e.g., the internet. Note that the disclosed 
hardware environment is exemplary. For example, production station 210 and stream 
server 220 can be implemented using two separate computer systems or using one computer 
system. In addition, if production station 210 and stream server 220 are implemented on 
separate computer systems as shown in FIG. 2, an optional direct connection (not shown) 
between production station 210 and stream server 22 0 can provide faster uploads of 
compressed video and annotation streams. In the following description, an audio stream 
optionally accompanies each video stream . 

Detailed Description Text (14) : 

A producer 215, installed in production station 210, is a user- friendly tool for use by 
a designer 219 to create a synchronization script which includes annotation stream(s) . 
The annotation stream(s) define the content (s) of a LiveScreen display 245 to be 
displayed on client computer 240 for a viewer 249. LiveScreen 245 display provides a 
graphical user interface (GUI) with multiple windows for synchronously displaying a 
video stream from stream server 22 0 and at least one displayable event stream. Examples 
of displayable events include textual/graphical information such as HTML-scripted web 
page(s) from web server 230. 

Detailed Description Text (15) : 

In one embodiment, as shown in FIG. 3, producer 215 includes a capture module 317 and 
an author module 318. Production station 210 includes 16 MB of RAM and a 1 GB hard disk 
drive for capturing and storing an uncompressed or precompressed video stream . Sources 
for generating video streams include a video camera 312, a video cassette recorder 
(VCR) (not shown) or a previously digitized video file 314, e.g., a Video for Windows 
(.avi) file. For ease of installation and use by designer 219, producer 215 is 
implemented in a host environment which includes a window-based operating system such 
as Microsoft Windows 95 and a web browser such as Netscape's Navigator 3.x. (Appendix A 
is a detailed user manual for one implementation of producer 215) . 

Detailed Description Text (16) : 

Referring also to the flowchart of FIG. 4A, in step 410 capture module 317 captures a 
live video/audio stream from video camera 312 or from the previously stored video file 
314. If video camera 312 provides an analog video stream, e.g., an NTSC signal, a 
hardware capture card (not shown) provides the required conversion from the analog 
video stream to a digitized video stream . Because temporary storage of uncompressed 
video data is memory intensive, some form of pre-compression can be used to reduce the 
memory storage requirement of the input video stream during capture step 410 and prior 
to compression step 420. 

Detailed Description Text (17) : 

In step 420, capture module 420 compresses the digitized video stream using a suitable 
compression technique. In this embodiment, depending on the bandwidth capacity of the 
connection provided by network 290 between stream server 220 and client computer 240, 
e.g., a POTS modem, ISDN or Ethernet, a suitable frame resolution and frame rate 
combination is selected. A compression algorithm based on the H263 standard (see 
co-pending applications VXT 702 and 718) is used for compressing lower bandwidth video 
streams, e.g., at less than 56 kbps . Alternatively, a Vxpress format ( see co-pending 
application VXT 712) is used for compressing higher bandwidth video streams . FIG. 5 
shows an exemplary format 50 0 for storing and delivering a compressed video stream . 

Detailed Description Text (21) : 

Designer 219 may view frames from video stream 50 0 displayed in video window 720 for 
referencing and selecting appropriate time stamps to use in generating annotation 
streams. Within video window 720, VCR function buttons, e.g., a rewind button 724, a 
play button 726 and a fast forward button 728, are available for designer 219 to 
quickly traverse video stream 500. Since video window 720 is provided as a convenience 
for designer 219, if designer 219 has prior knowledge of the content of the video 
stream, designer 219 may proceed with the generation of the annotation streams without 
viewing video window 720. 

Detailed Description Text (27) : 

In accordance with another aspect of the invention, LiveScreen display 600 also 
includes a table of contents (TOO 630, enabling viewer 249 at client computer 240 to 
skip forward or backward to a point within the entire video/audio stream 500. TOC 630 
include one or more content labels, each indexed to a corresponding time stamp in video 
stream 500, as defined by TOC time markers 791, 792, 793, 794 in LiveScreen display 



3 of 10 



9/12/03 9:36 PM 

I 



Record Display Form 



http://westbrs:8002ftin/gatexxe?f=TOC^ 



600. 

Detailed Description Text (28) : 

Referring now to FIG. 9, in one embodiment of the present invention, client computer 
240 includes a web browser 950 and a browser plug-in module 952 for interfacing web 
browser 950 with a main client module 960 . Client module 960 includes an event registry 
962, playout buffer (s) 966, video/audio decoder (s) 964, video /audio renderer(s) 965 and 
one or more dynamically loadable event applet(s), e.g., flipper applet 967, ticker 
applet 968 and VCR applet 969. In this embodiment, event registry 962 also functions as 
an annotation interpreter 963 . 

Detailed Description Text (29) : 

FIG. 10A is a flowchart illustrating the operation of client module 960. Assume that 
viewer 249 has not previously loaded client module 960 in client computer 240, but has 
already loaded a web browser 950, e.g., Netscape's Navigator (step 1010). Viewer 249 
surfs the world-wide web (www) via the internet and locates a web site of interest to 
viewer 249. Typically, the web site of interest is hosted on web server 230. 
Accordingly, a target web page is downloaded from web server 230 and displayed on 
client computer 240 . 

Detailed Description Text (30) : 

The target web page includes a link to a customized LiveScreen display, e.g., display 
600. If client module 960 has not been previously loaded, client module 960 is now 
loaded over web browser 950 for processing video/audio and annotation streams (step 
102 0) . Depending on the implementation, a copy of client module 960 may be available 
from the web site of interest. Alternatively, the target web page may provide a HTML 
link to another web server which has an updated copy of client module 960. 

Detailed Description Text (31) : 

Referring now to FIG. 10B, first, browser plug-in module 952 is installed over web 
browser 950 (step 1022) . As discussed above, plug-in module 952 provides the interface 
between client module 960 and web browser 950. The target web page provides a HTML link 
to the format for LiveScreen display 600. The LiveScreen display format is retrieved 
and display 600 is installed on client computer 240 using web browser 950 (step 1024) . 

Detailed Description Text (33) : 

Referring back to FIG. 10A, encoded video/audio frames and associated annotation frames 
are streamed from stream server 22 0 to client computer 240 for synchronous display 
(step 1030) . Streaming video and audio streams over a network is very efficient because 
streaming eliminates the need for a large buffer at client computer 240. In addition, 
streaming also provides flexibility, e.g., switching video sources midstream is 
possible without wasting network resources since streaming is based on a pseudo 
just-in-time (JIT) protocol and does not involve downloads of the entire video stream 
prior to display at client computer 240. If the underlying transmission protocol is 
HTTP, then video, audio and annotation packets are initially "pulled" by client 
computer 240 from server 220 using HTTP "get" packet (s) . 

Detailed Description Text (34) : 

Next, the encoded video/audio streams are decoded by decoder 964, i.e., decompressed 
using a suitable technique, and then displayed at client computer 240 by renderer 965 
(step 1040). (See co-pending applications VXT 702, 712 and 718). 

Detailed Description Text (37) : 

Further, since the video and annotation streams are handled synchronously but 
separately by video decoder 964 and annotation interpreter 963, respectively, steps 
1040 and 1050 can occur concurrently or consecutively. As discussed above, event 
registry 962 is capable of dynamic registration of event applets. Accordingly, 
annotation interpreter 963 is adaptable, and capable of automatic installation and 
linking of new event applet (s) to add new class (es) of displayable events for client 
computer 240. 

Detailed Description Text (38) : 

After registering with event registry 962, flipper applet 967 provides the location of 
the flipper stream to browser 950 which then begin receiving the flipper stream from 
stream server 220. Flipper annotation frames are provided by stream server 220 
synchronously with the video/audio frames to client module 960 so that the annotations, 
i.e., displayable events can be synchronized for display at client computer 240 (step 
1060) . In this example, URL addresses, for synchronizing HTML page flips with video 
stream are provided to web browser 950 thereby permitting client computer 240 to 
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subsequently retrieve and display various textual and graphical elements changing at 
predetermined points corresponding to the timeline of the video stream . Note that HTML 
pages can be retrieved from one or more web server (s) 230. 

Detailed Description Text (39) : 

Similarly, after registering with event registry 962, ticker (tape) applet 968 provides 
the location of the ticker stream to browser 950 which then begins receiving the ticker 
stream from stream server 220 . Ticker annotation frames are provided by stream server 
220 synchronously with the video/audio frames so that the annotations, i.e., 
displayable ticker data can be synchronized for display at client computer 240 at 
predetermined points corresponding to the timeline of the video stream . 

Detailed Description Text (42) : 

As shown in FIG. 11, a table of content 630 with content labels enables viewer 24 9 to 
skip forward or backward to predetermined locations in the video/audio stream. First, 
viewer 249 selects a content label of interest (step 1110) . Examples of suitable 
content labels are section headings of the video stream . Next, client module 960 sends 
a message to stream server 220 with the time stamp of an I -frame from the video stream 
whose location is close to selected content label (step 1120) . In this embodiment, an 
I-frame is a video frame which includes data for a complete video frame. Although 
computationally more intensive, it is also possible to select a P-frame and then 
reconstructed a complete video starting from a neighboring I-frame close to the 
selected P-frame. 

CLAIMS : 

1. In a client computer having a processor, memory and a display device, said client 
computer coupled to a stream server and at least one web server via a network, a method 
for synchronizing the display of video frames of a video stream with the display of 
displayable events, said method comprising the steps of: 

receiving a plurality of video frames of said video stream from said stream server for 
display in a video window of said display device, each said video frame associated with 
a time stamp; 

displaying said video frames in said video window; 

receiving a first plurality of annotation frames of a first annotation stream from said 
stream server; 

interpreting said first plurality of annotation frames which is associated with a first 
selected subset of said video frames, each of said first plurality of annotation frames 
including an event locator and an event time marker, and wherein said event time 
markers of said first plurality of annotation frames corresponds to the respective time 
stamps of said first selected subset of video frames, and each said event locator 
points to one of a first plurality of displayable events stored in said at least one 
web server; and o 

retrieving and synchronously displaying in a first event window of said display device 
the displayable event pointed to by the event locator of each of said first plurality 
of annotation frames whenever the event time marker of each of said first plurality of 
annotation frames matches the time stamp associated with the video frame currently 
being displayed in said video window. 

4. The method of claim 1 wherein said video server and said web server are part of a 
server computer system. 

6. In a stream server having a processor and memory, said stream server coupled to a 
client computer and at least one web server via a network, said client computer having 
a processor, memory and a display device, a method for streaming video and annotation 
streams to synchronize the display of video frames of a video stream with the display 
of displayable events, said method comprising the steps of: 

streaming a plurality of video frames of said video stream from said stream server to 
said client computer for display in a video window of said display device, each said 
video frame associated with a time stamp; and 



synchronously streaming a first plurality of annotation frames of a first annotation 
stream from said stream server to said client computer, said first plurality of 
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annotation frames associated with a first selected subset of said video frames, each of 
said first plurality of annotation frames including an event locator and an event time 
marker, and wherein each said event locator points to one of a first plurality of 
displayable events stored in said at least one web server, said event time markers of 
said first plurality of annotation frames corresponds to the respective time stamps of 
said first selected subset of video frames, and said video stream and said first 
annotation stream are synchronized using said respective time stamps of said first 
selected subset of said video frames and said event time markers of said first 
plurality of annotation frames. 

8. The method of claim 6 wherein comprising the step of: 

synchronously streaming a second plurality of annotation frames of a second annotation 
stream from said stream server to said client computer, said second plurality of 
annotation frames associated with a second selected subset of said video frames, each 
of said second plurality of annotation frames including a corresponding one of a second 
plurality of displayable events and an event time marker, and wherein said event time 
markers of said second plurality of annotation frames corresponds to the respective 
time stamps of said second selected subset of video frames, and said video stream and 
said annotation stream are synchronized using said respective time stamps of said 
second selected subset of said video frames and said event time markers of said second 
plurality of annotation frames. 

10. In a client computer having a processor, memory and a display device, said client 
computer coupled to a stream server via a network, a method for synchronizing the 
display of video frames of a video stream and the display of a plurality of displayable 
events, wherein said video stream and said plurality of displayable events are to be 
displayed concurrently in a video window and an event window, respectively, of said 
display device, said method comprising the steps of: 

receiving a plurality of video frames of said video stream from said stream server for 
display in said video window, each said video frame associated with a time stamp; 

displaying said video frames in said video window; 

receiving a plurality of annotation frames of an annotation stream from said stream 
server; 



interpreting said plurality of annotation frames, said plurality of annotation frames 
associated with a selected subset of said video frames, each of said plurality of 
annotation frames including a corresponding one of said displayable events and an event 
time marker, and wherein said event time markers of said plurality of annotation frames 
corresponds to the respective time stamps of said selected subset of video frames; and 

synchronously displaying in said event window said one corresponding displayable event 
whenever the event time marker of each of said plurality of annotation frames matches 
the time stamp associated with the video frame currently being displayed in said video 
window. 



12 . In a stream server having a processor and memory, said stream server coupled to a 
client computer via a network, said client computer having a processor, memory and a 
display device, a method for streaming video and annotation streams to synchronize the 
display of video frames of said video stream with the display of a plurality of 
displayable events, wherein said video stream and said plurality of displayable events 
are to be displayed concurrently in a video window and a first event window, 
respectively, of said display device, said method comprising the steps of: 

streaming a plurality of video frames of said video stream from said stream server to 
said client computer for display in said video window, each said video frame associated 
with a time stamp; and 

synchronously streaming a plurality of annotation frames of an annotation stream from 
said stream server to said client computer, said plurality of annotation frames 
associated with a selected subset of said video frames, each of said plurality of 
annotation frames including a corresponding one of said displayable events and an event 
time marker, and wherein said event time markers of said plurality of annotation frames 
corresponds to the respective time stamps of said selected subset of video frames, and 
said video stream and said annotation stream are synchronized using said respective 
time stamps and said event time markers. 



6ofl0 



9/12/03 9:36 PM 



Record Display Form http://westbrs:8002^in/gate.exe?f=TO 



14. A client computer useful in association with a stream server and at least one web 
server, said client computer comprising: 

a playout buffer configured to receive a plurality of video frames of a video stream 
from said stream server, each said video frame associated with a time stamp; 

an annotation interpreter configured to receive and interpret a first plurality of 
annotation frames of a first annotation stream from said stream server, said first 
plurality of annotation frames associated with a first selected subset of said video 
frames, each of said first plurality of annotation frames including an event locator 
and an event time marker, and wherein said event time markers of said first plurality 
of annotation frames corresponds to the respective time stamps of said first selected 
subset of video frames, and each said event locator points to one of a first plurality 
of displayable events stored in said at least one web server; and 

a display device configured to display said video frames in a video window of said 
display device and further configured to synchronously display in a first event window 
of said display device the displayable event pointed to by the event locator of each of 
said first plurality of annotation frames whenever the event time marker of each of 
said first plurality of annotation frames matches the time stamp associated with the 
video frame currently being displayed in said video window. 

17. The client computer of claim 14 wherein said video server and said web server are 
part of a server computer system. 

19. A stream server useful in association with a client computer and at least one web 
server, said client computer having a display device, the stream server comprising"! 

memory configured to store a plurality of video frames of said video stream to be 
streamed from said stream server to said client computer for display in a video window 
of said display device, each said video frame associated with a time stamp; and 

wherein said memory is further configured to store a first plurality of annotation 
frames of a first annotation stream to be synchronously streamed from said stream 
server to said client computer, said first plurality of annotation frames associated 
with a first selected subset of said video frames, each of said first plurality of 
annotation frames including an event locator and an event time marker, and wherein each 
said event locator points to one of a first plurality of displayable events stored in 
said at least one web server, said event time markers of said first plurality of 
annotation frames corresponds to the respective time stamps of said first selected 
subset of video frames, and said video stream and said first annotation stream are 
synchronized using said respective time stamps of said first selected subset of said 
video frames and said event time markers of said first plurality of annotation frames. 

21. The stream server of claim 19 wherein said memory is further configured to store a 
second plurality of annotation frames of a second annotation stream to be synchronously 
streamed from said stream server to said client computer, said second plurality of 
annotation frames associated with a second selected subset of said video frames, each 
of said second plurality of annotation frames including a corresponding one of a second 
plurality of displayable events and an event time marker, and wherein said event time 
markers of said second plurality of annotation frames corresponds to the respective 
time stamps of said second selected subset of video frames, and said video stream and 
said annotation stream are synchronized using said respective time stamps of said 
second selected subset of said video frames and said event time markers of said second 
plurality of annotation frames. 

23. A client computer useful in association with a stream server, said client computer 
comprising : 

a playout buffer configured to receive a plurality of video frames of said video stream 
from said stream server, each said video frame associated with a time stamp; 

an annotation interpreter configured to receive and interpret a plurality of annotation 
frames of an annotation stream from said stream server, said plurality of annotation 
frames associated with a selected subset of said video frames, each of said plurality 
of annotation frames including a corresponding one of said displayable events and an 
event time marker, and wherein said event time markers of said plurality of annotation 
frames corresponds to the respective time stamps of said selected subset of video 
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a display device configured to display said video frames in a video window of said 
display device, and further configured to synchronously display in an event window of 
said display device said one corresponding displayable event whenever the event time 
marker of each of said plurality of annotation frames matches the time stamp associated 
with the video frame currently being displayed in said video window. 

25. A stream server useful in association with a client computer, said client computer 
having a display device, the stream server comprising: 

memory configured to store a plurality of video frames of said video stream to be 
streamed from said stream server to said client computer for display in a video window 
of said display device, each said video frame associated with a time stamp; and 

wherein said memory is further configured to store a plurality of annotation frames of 
an annotation stream to be synchronously streamed from said stream server to said 
client computer, said plurality of annotation frames associated with a selected subset 
of said video frames, each of said plurality of annotation frames including a 
corresponding one of a plurality of displayable events and an event time marker, and 
wherein said event time markers of said plurality of annotation frames corresponds to 
the respective time stamps of said selected subset of video frames, and said video 
stream and said annotation stream are synchronized using said respective time stamps of 
said selected subset of said video frames and said event time markers of said plurality 
of annotation frames. 

27. A computer-readable medium useful in association with a client computer having a 
processor, memory and a display device, said client computer coupled to a stream server 
and at least one web server via a network, the computer- readable medium comprising 
computer -readable code instructions configured to cause said client computer to execute 
the steps of : 

receiving a plurality of video frames of a video stream from said stream server for 
display in a video window of said display device, each said video frame associated with 
a time stamp; 

displaying said video frames in said video window; 

receiving a first plurality of annotation frames of a first annotation stream from said 
stream server; 

interpreting said first plurality of annotation frames which is associated with a first 
selected subset of said video frames, each of said first plurality of annotation frames 
including an event locator and an event time marker, and wherein said event time 
markers of said first plurality of annotation frames corresponds to the respective time 
stamps of said first selected subset of video frames, and each said event locator 
points to one of a first plurality of displayable events stored in said at least one 
web server; and 

retrieving and synchronously displaying in a first event window of said display device 
the displayable event pointed to by the event locator of each of said first plurality 
of annotation frames whenever the event time marker of each of said first plurality of 
annotation frames matches the time stamp associated with the video frame currently 
being displayed in said video window. 

30. The computer-readable medium of claim 27 wherein said video server and said web 
server are part of a server computer system. 

32. A computer-readable medium useful in association with a stream server having a 
processor and memory, said stream server coupled to a client computer and at least one 
web server via a network, said client computer having a processor, memory and a display 
device, the computer- readable medium comprising computer- readable code instructions 
configured to cause said stream server to execute the steps of: 

streaming a plurality of video frames of a video stream from said stream server to said 
client computer for display in a video window of said display device, each said video 
frame associated with a time stamp; and 

synchronously streaming a first plurality of annotation frames of a first annotation 
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stream from said stream server to said client computer, said first plurality of 
annotation frames associated with a first selected subset of said video frames, each of 
said first plurality of annotation frames including an event locator and an event time 
marker, and wherein each said event locator points to one of a first plurality of 
displayable events stored in said at least one web server, said event time markers of 
said first plurality of annotation frames corresponds to the respective time stamps of 
said first selected subset of video frames, and said video stream and said first 
annotation stream are synchronized using said respective time stamps of said first 
selected subset of said video frames and said event time markers of said first 
plurality of annotation frames. 

34. The computer-readable medium of claim 32 further comprising computer-readable code 
instructions configured to cause said stream server to execute the step of : 

synchronously streaming a second plurality of annotation frames of a second annotation 
stream from said stream server to said client computer, said second plurality of 
annotation frames associated with a second selected subset of said video frames, each 
of said second plurality of annotation frames including a corresponding one of a second 
plurality of displayable events and an event time marker, and wherein said event time 
markers of said second plurality of annotation frames corresponds to the respective 
time stamps of said second selected subset of video frames, and said video stream and 
said annotation stream are synchronized using said respective time stamps of said 
second selected subset of said video frames and said event time markers of said second 
plurality of annotation frames. 

36. A computer- readable medium useful in association with a client computer having a 
processor, memory and a display device, said client computer coupled to a stream server 
via a network, the computer- readable medium comprising computer- readable code 
instructions configured to cause said client computer to execute the steps of: 

receiving a plurality of video frames of a video stream from said stream server for 
display in a video window of said display device, each said video frame associated with 
a time stamp; 

displaying said video frames in said video window; 

receiving a plurality of annotation frames of an annotation stream from said stream 
server; 



interpreting said plurality of annotation frames, said plurality of annotation frames 
associated with a selected subset of said video frames, each of said plurality of 
annotation frames including a corresponding one of a plurality of displayable events 
and an event time marker, and wherein said event time markers of said plurality of 
annotation frames corresponds to the respective time stamps of said selected subset of 
video frames; and 

synchronously displaying in an event window of said display device said one 
corresponding displayable event whenever the event time marker of each of said 
plurality of annotation frames matches the time stamp associated with the video frame 
currently being displayed in said video window. 

38. A computer-readable medium useful in association with a stream server having a 
processor and memory, said stream server coupled to a client computer via a network, 
said client computer having a processor, memory and a display device, the 
computer- readable medium comprising computer-readable code instructions configured to 
cause said stream server to execute the steps of : 

streaming a plurality of video frames of a video stream from said stream server to said 
client computer for display in a video window of said display device, each said video 
frame associated with a time stamp; and 

synchronously streaming a plurality of annotation frames of an annotation stream from 
said stream server to said client computer, said plurality of annotation frames 
associated with a selected subset of said video frames, each of said plurality of 
annotation frames including a corresponding one of said displayable events and an event 
time marker, and wherein said event time markers of said plurality of annotation frames 
corresponds to the respective time stamps of said selected subset of video frames, and 
said video stream and said annotation stream are synchronized using said respective 
time stamps and said event time markers. 
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The production of synchronization scripts and associated annotated multimedia streams 
for servers and client computers coupled to each other by a diverse computer network 
which includes local area networks (LANs) and/or wide area networks (WANs) such as the 
intermet. Annotated multimedia streams can include a compressed video stream for 
display in a video window, an accompanying compressed audio stream and annotations. 
Synchronization scripts include annotation streams for synchronizing the display of 
video streams with annotations, e.g., displayable events, such textual/graphical data 
in the form of HTML pages with Java applets to be displayed in one or more event 
windows. The producer includes a capture module and an author module for capturing 
video streams and generating annotation streams, respectively. The capture module 
compresses the video stream using a suitable compression format. Annotation streams 
include annotation frames which provide either pointer (s) to the event (s) of interest 
or include displayable data embedded within the annotation stream. Accordingly, each 
annotation frame includes either an event locator or an event data. In addition, each 
annotation frame includes an event time marker which corresponds to the time stamp (s) 
of associated video frame (s) within the video stream . Embedded displayable data include 
ticker tape data embedded within the annotation stream. Examples of event locators to 
displayable events include URL addresses pointing to HTML web pages. The video/audio 
streams and annotation streams are stored in stream server (s) for subsequent retrieval 
by client computer (s) in a coordinated manner, so that the client computer (s) is able 
to synchronously display the video frames and displayable event (s) in a video window 
and event window (s) , respectively. In one implementation, annotation streams include a 
flipper stream for locating HTML pages and a ticker stream which include ticker (tape) 
data . 
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application Ser. No. 08/818,804, filed on Mar. 14, 1997, entitled "Production of a 
Video Stream with Synchronized Annotations over a Computer Network, U.S. application 
Ser. No. 08/819,586, filed on Mar. 14, 1997, entitled "Method and Apparatus for 
Implementing Control Functions in a Streamed Video Display System, U.S. application 
Ser. No. 08/818,769, filed on Mar. 14, 1997, entitled "Method and Apparatus for 
Automatically Detecting Protocols in a Computer Network", U.S. application Ser. No. 
08/818,127, filed on Mar. 14, 1997, entitled "Dynamic Bandwidth Selection for Efficient 
Transmission of Multimedia Streams in a Computer Network" , U.S. application Ser. No. 
08/819,585, filed on Mar. 14, 1997, entitled "Streaming and Display of a Video Stream 
with Synchronized Annotations over a Computer Network", U.S. application Ser. No. 
08/818,664, filed on Mar. 14, 1997, entitled "Selective Retransmission for Efficient 
and Reliable Streaming of Multimedia Packets in a Computer Network", U.S. application 
Ser. No. 08/819,579, filed on Mar. 14, 1997, entitled "Method and Apparatus for 
Table-Based Compression with Embedded Coding", U.S. application Ser. No. 08/819,587, 
filed on Mar. 14, 1997, entitled "Method and Apparatus for Implementing Motion 
Estimation in Video Compression", U.S. application Ser. No. 08/818,826, filed on Mar. 
14, 1997, entitled "Digital Video Signal Encoder and Encoding Method", all filed 
concurrently herewith, U.S. application Ser. No. 08/822,156, filed on Mar. 17, 1997, 
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entitled "Method and Apparatus for Communication Media Commands and Data Using the HTTP 
Protocol", provisional U.S. Application Serial No. 60/036,662, filed on Jan. 30, 1997, 
entitled "Methods and Apparatus for Autodetecting Protocols in a Computer Network" U.S. 
application Ser. No. 08/625,650, filed on Mar. 29, 1996, entitled "Table-Based 
Low-Level Image Classification System", U.S. application Ser. No. 08/714,447, filed on 
Sep. 16, 1996, entitled "Multimedia Compression System with Additive Temporal Layers", 
and is a continuation-in-part of U.S. application Ser. No. 08/623,299, filed on Mar. 
28, 1996, entitled "Table-Based Compression with Embedded Coding", which are all 
incorporated by reference in their entirety for all purposes. 

Brief Summary Text (6) : 

Existing conventional internet applications, such as electronic mailers and web 
browser, are capable of transferring and presenting textual and graphical information. 
However, none of these individual internet applications effectively provide synchronous 
delivery of a combination of diverse multimedia streams in a coherent and integrated 
manner. This is because executing several independent and unrelated applications to 
present the diverse combination of multimedia streams on a client computer can result 
in a hodgepodge of poor quality, incompatible and/or incoherent presentations. 

Brief Summary Text (7) : 

In view of the foregoing, there are desired improved techniques for reliably providing 
a multimedia stream such as a video and audio stream, together with annotations such as 
textual and graphical information in an integrated seamless package to client 
computer(s), while efficiently utilizing the network resources and consuming minimal 
computational cycles on the client computer(s) . 

Brief Summary Text (9) : 

The present invention provides synchronization scripts and associated annotated 
multimedia streams for servers and client computers coupled to each other by a diverse 
computer network which includes local area networks (LANs) and/or wide area networks 
(WANs) such as the internet. Annotated multimedia streams can include a compressed 
video stream for display in a video window, an accompanying compressed audio stream and 
annotations. Synchronization scripts include annotation streams for synchronizing the 
display of video streams with annotations, e.g., displayable events, such 
textual/graphical data in the form of HTML pages with Java applets to be displayed in 
one or more event windows. 

Brief Summary Text (10) : 

In one embodiment, a producer includes a capture module and an author module for 
capturing video streams and generating annotation streams, respectively. The video and 
annotation streams are then stored in stream server (s) to be provided to one or more 
client computer (s) upon request. 

Brief Summary Text (11) : 

The capture module compresses the video stream using a compression format based on a 
standard H263, generating, for example, a QCIF resolution (176 . times . 144) video frames 
at 10-20 frames per second (fps) which can be encoded and transmitted over a 20 Kbps 
connection. Alternatively, using the scalable vector quantization (SVQ) compression 
algorithm of the present invention, dynamically scalable data transmission rates from 
10Kbps to several Mbps can support scalable resolutions of 160 . times . 120 to 
640 . times. 480 and frames rates ranging from 1 fps to 30 fps. Other compression 
techniques can also be used with the present invention. 

Brief Summary Text (12) : 

In this embodiment, annotation streams include annotation frames which provide either 
pointer (s) to the event (s) of interest or include displayable data embedded within the 
annotation stream. Accordingly, each annotation frame includes either an event locator 
or an event data. In addition, each annotation frame includes an event time marker 
which corresponds to the time stamp (s) of associated video frame (s) within the video 
stream . Examples of embedded displayable data include ticker tape data embedded within 
the annotation stream. Examples of event locators to displayable events include URL 
addresses pointing to HTML web pages. Note that an event time marker need not be 
identical to a corresponding video time stamp. The client computer is capable of 
switching to a new displayable event together with a video frame or in between two 
video frames. 

Drawing Description Text (5) : 

FIG. 4A is a flowchart illustrating the capture of a live video/audio stream from a 
video camera or from a previously stored video file. 
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Drawing Description Text (7) : 

FIG. 5 shows an exemplary format for storing and delivering a compressed video stream . 
Drawing Description Text (11) : 

FIG. 9 illustrates one embodiment of the client computer which includes a web browser 
and a browser plug-in module for interfacing a web browser with a client module . 

Detailed Description Text (13) : 

FIG. 2 is a block diagram showing an exemplary hardware environment for practicing the 
annotated video-on-demand (VOD) system of the present invention. The VOD system 
includes a production station 210, a stream server 220, at least one web server 230 and 
at least one client computer 240, each of which can be implemented using computer 
system 10 0 described above. Stream server 22 0 and web server 230 are coupled to client 
computer 240 via a computer network 290, e.g., the internet. Note that the disclosed 
hardware environment is exemplary. For example, production station 210 and stream 
server 220 can be implemented using two separate computer systems or using one computer 
system. In addition, if production station 210 and stream server 220 are implemented on 
separate computer systems as shown in FIG. 2, an optional direct connection (not shown) 
between production station 210 and stream server 22 0 can provide faster uploads of 
compressed video and annotation streams. In the following description, an audio stream 
optionally accompanies each video stream . 

Detailed Description Text (14) : 

A producer 215, installed in production station 210, is a user-friendly tool for use by 
a designer 219 to create a synchronization script which includes annotation stream(s) . 
The annotation stream(s) define the content (s) of a LiveScreen display 245 to be 
displayed on client computer 240 for a viewer 249. LiveScreen 245 display provides a 
graphical user interface (GUI) with multiple windows for synchronously displaying a 
video stream from stream server 22 0 and at least one displayable event stream. Examples 
of displayable events include textual/graphical information such as HTML- -scripted web 
page(s) from web server 230. 

Detailed Description Text (15) : 

In one embodiment, as shown in FIG. 3, producer 215 includes a capture module 317 and 
an author module 318. Production station 210 includes 16 MB of RAM and a 1 GB hard disk 
drive for capturing and storing an uncompressed or precompressed video stream . Sources 
for generating video streams include a video camera 312, a video cassette recorder 
(VCR) (not shown) or a previously digitized video file 314, e.g., a Video for Windows 
(.avi) file. For ease of installation and use by designer 219, producer 215 is 
implemented in a host environment which includes a window-based operating system such 
as Microsoft Windows 95 and a web browser such as Netscape's Navigator 3.x. (Appendix A 
is a detailed user manual for one implementation of producer 215) . 

Detailed Description Text (16) : 

Referring also to the flowchart of FIG. 4A, in step 410 capture module 317 captures a 
live video/audio stream from video camera 312 or from the previously stored video file 
314. If video camera 312 provides an analog video stream, e.g., an NTSC signal, a 
hardware capture card (not shown) provides the required conversion from the analog 
video stream to a digitized video stream . Because temporary storage of uncompressed 
video data is memory intensive, some form of pre-compression can be used to reduce the 
memory storage requirement of the input video stream during capture step 410 and prior 
to compression step 420. 

Detailed Description Text (17) : 

In step 420, capture module 420 compresses the digitized video stream using a suitable 
compression technique. In this embodiment, depending on the bandwidth capacity of the 
connection provided by network 290 between stream server 220 and client computer 240, 
e.g., a POTS modem, ISDN or Ethernet, a suitable frame resolution and frame rate 
combination is selected. A compression algorithm based on the H263 standard (see 
co-pending applications VXT 702 and 718) is used for compressing lower bandwidth video 
streams, e.g., at less than 56 kbps . Alteratively, a Vxpress format (see co-pending 
application VXT 712) is used for compressing higher bandwidth video streams . FIG. 5 
shows an exemplary format 500 for storing and delivering a compressed video stream . 

Detailed Description Text (21) : 

Designer 219 may view frames from video stream 500 displayed in video window 720 for 
referencing and selecting appropriate time stamps to use in generating annotation 
streams. Within video window 720, VCR function buttons, e.g., a rewind button 724, a 
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play button 726 and a fast forward button 728, are available for designer 219 to 
quickly traverse video stream 500. Since video window 720 is provided as a convenience 
for designer 219, if designer 219 has prior knowledge of the content of the video 
stream, designer 219 may proceed with the generation of the annotation streams without 
viewing video window 720. 

Detailed Description Text (27) : 

In accordance with another aspect of the invention, LiveScreen display 600 also 
includes a table of contents (TOC) 630, enabling viewer 249 at client computer 240 to 
skip forward or backward to a point within the entire video/audio stream 500. TOC 630 
include one or more content labels, each indexed to a corresponding time stamp in video 
stream 500, as defined by TOC time markers 791, 792, 793, 794 in LiveScreen display 
600. 



Detailed Description Text (28) : 

Referring now to FIG. 9, in one embodiment of the present invention, client computer 
240 includes a web browser 950 and a browser plug- in module 952 for interfacing web 
browser 950 with a main client module 960. Client module 960 includes an event registry 
962, playout buffer (s) 966, video/audio decoder (s) 964, video /audio renderer(s) 965 and 
one or more dynamically loadable event applet(s), e.g., flipper applet 967, ticker 
applet 968 and VCR applet 96 9. In this embodiment, event registry 962 also functions as 
an annotation interpreter 963 . 

Detailed Description Text (29) : 

FIG. 10A is a flowchart illustrating the operation of client module 960. Assume that 
viewer 249 has not previously loaded client module 960 in client computer 240, but has 
already loaded a web browser 950, e.g., Netscape's Navigator (step 1010). Viewer 249 
surfs the world-wide web (www) via the internet and locates a web site of interest to 
viewer 249. Typically, the web site of interest is hosted on web server 230. 
Accordingly, a target web page is downloaded from web server 230 and displayed on 
client computer 240. 

Detailed Description Text (30) : 

The target web page includes a link to a customized LiveScreen display, e.g., display 
600. If client module 960 has not been previously loaded, client module 960 is now 
loaded over web browser 950 for processing video/audio and annotation streams (step 
1020) . Depending on the implementation, a copy of client module 960 may be available 
from the web site of interest. Alternatively, the target web page may provide a HTML 
link to another web server which has an updated copy of client module 960 . 

Detailed Description Text (31): 

Referring now to FIG. 10B, first, browser plug-in module 952 is installed over web 
browser 950 (step 1022) . As discussed above, plug-in module 952 provides the interface 
between client module 960 and web browser 950. The target web page provides a HTML link 
to the format for LiveScreen display 600. The LiveScreen display format is retrieved 
and display 600 is installed on client computer 240 using web browser 950 (step 1024) . 

Detailed Description Text (33) : 

Referring back to FIG. 10A, encoded video/audio frames and associated annotation frames 
are streamed from stream server 220 to client computer 240 for synchronous display 
(step 1030). Streaming video and audio streams over a network is very efficient because 
streaming eliminates the need for a large buffer at client computer 240. In addition, 
streaming also provides flexibility, e.g., switching video sources midstream is 
possible without wasting network resources since streaming is based on a pseudo 
just-in-time (JIT) protocol and does not involve downloads of the entire video stream 
prior to display at client computer 240. If the underlying transmission protocol is 
HTTP, then video, audio and annotation packets are initially "pulled" by client 
computer 240 from server 220 using HTML "get" packet (s) . 

Detailed Description Text (34) : 

Next, the encoded video/audio streams are decoded by decoder 964, i.e., decompressed 
using a suitable technique, and then displayed at client computer 240 by renderer 965 
(step 1040) . (See co-pending applications VXT 702, 712 and 718) . 

Detailed Description Text (37) : 

Further, since the video and annotation streams are handled synchronously but 
separately by video decoder 964 and annotation interpreter 963, respectively, steps 
1040 and 1050 can occur concurrently or consecutively. As discussed above, event 
registry 962 is capable of dynamic registration of event applets. Accordingly, 
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annotation interpreter 963 is adaptable, and capable of automatic installation and 
linking of new event applet (s) to add new class (es) of displayable events for client 
computer 240 . 

Detailed Description Text (38) : 

After registering with event registry 962, flipper applet 967 provides the location of 
the flipper stream to browser 950 which then begin receiving the flipper steam from 
stream server 220. Flipper annotation frames are provided by stream server 220 
synchronously with the video/audio frames to client module 960 so that the annotations, 
i.e., displayable events can be synchronized for display at client computer 240 (step 
1060) . In this example, URL addresses, for synchronizing HTML page flips with video 
stream are provided to web browser 950 thereby permitting client computer 240 to 
subsequently retrieve and display various textual and graphical elements changing at 
predetermined points corresponding to the timeline of the video stream . Note that HTML 
pages can be retrieved from one or more web server (s) 230. 

Detailed Description Text (39) : 

Similarly, after registering with event registry 962, ticker (tape) applet 968 provides 
the location of the ticker stream to browser 950 which then begins receiving the ticker 
stream from stream server 220. Ticker annotation frames are provided by stream server 
220 synchronously with the video/audio frames so that the annotations, i.e., 
displayable ticker data can be synchronized for display at client computer 240 at 
predetermined points corresponding to the timeline of the video stream . 

Detailed Description Text (42) : 

As shown in FIG. 11, a table of content 630 with content labels enables viewer 249 to 
skip forward or backward to predetermined locations in the video/audio stream. First, 
viewer 249 selects a content label of interest (step 1110) . Examples of suitable 
content labels are section headings of the video stream . Next, client module 960 sends 
a message to stream server 220 with the time stamp of an I-frame from the video stream 
whose location is close to selected content label (step 1120) . In this embodiment, an 
I-frame is a video frame which includes data for a complete video frame. Although 
computationally more intensive, it is also possible to select a P-frame and then 
reconstructed a complete video starting from a neighboring I-frame close to the 
selected P-frame. 

Other Reference Publication (1) : 

"Web Theater Producer User Guide, Version 2.0", Palo Alto, CA: VXtreme, Inc., (1997). 
CLAIMS: 

1. A computer -readable medium useful in association with a client computer having a 
display device, the computer -readable medium having computer -executable instructions 
which cause the computer system to perform a method comprising: 

receiving a video stream from a stream server coupled to the client computer via a 
computer network, said video stream including a plurality of video frames, each said 
video frame including a time stamp; 

sequentially displaying said video frames on said display device, starting from an 
initial video frame of said video stream ; 

selecting a content label from a table of contents displayed on said display device, 
said content label providing an index into an intermediate video frame of said video 
stream, said index based on the time stamp of said intermediate video frame; 

communicating said index to said stream server; 

receiving said video stream from said stream server, starting from said intermediate 
video frame; and 

sequentially displaying said video frames on said display device, starting with said 
intermediate video frame. 

2. A computer-readable medium useful in association with a stream server, said stream 
server coupled to a client computer via a computer network, said client computer 
including a display device, the computer-readable medium having computer- executable 
instructions which cause the stream server to perform a method comprising: 
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streaming said video stream to said client computer for sequential display on said 
display device, starting from an initial video frame of a plurality of video frames of 
said video stream, each said video frame including a time stamp; 

receiving an index from said client computer, said index associated with a selectable 
content label from a table of contents displayed on said display device, said index 
based on the time stamp of an intermediate video frame of said video stream ; and 

streaming said video stream to said client computer for sequential display on said 
display device, starting from said intermediate video frame. 

15. A computer- readable medium useful in association with a client computer having a 
display device, the computer-readable medium comprising: 

receiving a video stream from a stream server coupled to the client computer via a 
computer network, said video stream including a plurality of video frames, each said 
video frame including a time stamp; 

sequentially displaying said video frames on said display device, starting from an 
initial video frame of said video stream ; 

selecting a content label from a table of contents displayed on said display device ; 

locating an I -frame in the video stream that is closest to the content label, wherein 
the I -frame has an associated timestamp; 

communicating said associated 'timestamp to said stream server; 

receiving said video stream from said stream server, starting from said intermediate 
video^ frame; and 

sequentially displaying said video frames on said display device, starting with said 
intermediate video frame. 

17. The client computer of claim 16, wherein the stream of data includes a video 
stream . 

18. The client computer of claim 17, wherein the stream of data further includes an 
annotation stream associated with the video stream. 
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TITLE: Dynamic bandwidth selection for efficient transmission of multimedia streams in 
a computer network 



Application Filing Date (1) : 
19970314 

Parent Case Text (2) : 

This application is related to U.S. application Ser. No. 08/818,805, filed on Mar. 14, 
1997, entitled "Method and Apparatus for Implementing Motion Detection in Video 
Compression, " U.S. application Ser. No. 08/819,507, filed Mar. 14, 1997, entitled 
"Digital Video Signal Encoder and Encoding Method," U.S. application Ser. No. 
08/818,804, filed on Mar. 14, 1997, entitled "Production of a Video Stream with 
Synchronized Annotations over a Computer Network," U.S. application Ser. No. 
08/819,586, filed on Mar. 14, 1997, entitled "Method and apparatus for Implementing 
Control Functions in a Streamed Video Display System," U.S. application Ser. No. 
08/818,769, filed on Mar. 14, 1997, entitled "Method and apparatus for Automatically 
Detecting Protocols in a Computer Network," U.S. application Ser. No. 08/818,127, filed 
on Mar. 14, 1997, entitled "Dynamic Bandwidth Selection for Efficient Transmission of 
Multimedia Streams in a Computer Network," U.S. application Ser. No. 08/819,585, filed 
on Mar. 14, 1997, entitled "Streaming and Display of a Video Stream with Synchronized 
Annotations over a Computer Network," U.S. application Ser. No. 0 8/818,664, filed on 
Mar. 14, 1997, entitled "Selective Retransmission for Efficient and Reliable Streaming 
of Multimedia Packets in a Computer Network," U.S application Ser. No. 08/819,579, 
filed Mar. 14, 1997, entitled "Method and apparatus for Table-Based Compression with 
Embedded Coding," U.S. application Ser. No. 08/819,587, filed Mar. 14, 1997, entitled 
"Method and apparatus for Implementing Motion Estimation in Video Compression," U.S. 
application Ser. No. 08/818,826, filed on Mar. 14, 1997, entitled "Digital Video Signal 
Encoder and Encoding Method," all filed concurrently herewith, U.S. application Ser. 
No. 08/822,156, filed on Mar. 17, 1997, entitled "Method and apparatus for 
Communication Media Commands and Data Using the HTTP Protocol," U.S. provisional 
application Serial No. 60/036, 662, filed on Jan. 30, 1997, entitled "Methods and 
apparatus for Autodetecting Protocols in a Computer Network," U.S. application Ser. No. 
08/625,650, filed on Mar. 29, 1996, entitled "Table-Based Low-Level Image 
Classification System," U.S. application Ser. No. 08/714,447, filed on Sep. 16, 1996, 
entitled "Multimedia Compression System with Additive Temporal Layers," and is a 
continuation-in-part of U.S. application Ser. No. 08/623,299, filed on Mar. 28, 1996, 
entitled "Table-Based Compression with Embedded Coding," which are all incorporated by 
reference in their entirety for all purposes. 

Brief Summary Text (13) : 

The present invention provides efficient transmission of multimedia streams from a 
server to a client computer over a diverse computer network including local area 
networks (LANs) and wide area networks (WANs) such as the internet. Examples of 
multimedia streams provided to the client computer include a compressed video stream, a 
compressed audio stream, and an annotation stream with pointers to t ext ua 1 / graphi ca 1 
data in the form of HTML pages. 

Drawing Description Text (4) : 

FIG. 3 is a block diagram showing a producer which includes a capture module and an 
author module for capturing video streams and for generating annotation streams, 
respectively. 

Detailed Description Text (13) : 

FIG. 2 is a block diagram showing an exemplary hardware environment for practicing the 
reliable and efficient video -on -demand (VOD) system of the present invention. The VOD 
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system includes a production station 210, a stream server 220, at least one web server 
230 and at least one client computer 240, each of which can be implemented using 
computer system 100 described above. Stream server 220 and web server 230 are coupled 
to client computer 240 via a computer network 290, e.g., the internet. Note that the 
disclosed hardware environment is exemplary. For example, production station 210 and 
stream server 220 can be implemented using two separate computer systems or using one 
computer system. In addition, if production station 210 and stream server 220 are 
implemented on separate computer systems as shown in FIG. 2, an optional direct 
connection (not shown) between production station 210 and stream server 220 can provide 
faster uploads of compressed video and annotation streams. In the following 
description, an audio stream optionally accompanies each video stream . 

Detailed Description Text (14) : 

A producer 215, installed in production station 210, is a user-friendly tool for use by 
a designer 219 to create a synchronization script which includes annotation stream(s) . 
The annotation stream(s) define the content (s) of a LiveScreen display 245 to be 
displayed on client computer 240 for a viewer 249. LiveScreen 245 display provides a 
graphical user interface (GUI) with multiple windows for synchronously displaying a 
video stream from stream server 220 and at least one displayable event stream. Examples 
of displayable events include textual/graphical information such as HTML-scripted web 
page(s) from web server 230. 

Detailed Description Text (15) : 

Referring to FIG. 3, producer 215 includes a capture module 317 and an author module 
318. Production station 210 includes 16 MB of RAM and a 1 GB hard disk drive for 
capturing and storing an uncompressed or precompressed video stream . Sources for 
generating video streams include a video camera 312, a video cassette recorder (VCR) 
(not shown) or a previously digitized video file 314, e.g., a Video for Windows (.avi) 
file. For ease of installation and use by designer 219, producer 215 is implemented in 
a host environment which includes a window-based operating system such as Microsoft 
Windows 95 and a web browser such as Netscape's Navigator 2.x. 

Detailed Description Text (16) : 

Client computer 240 in FIG. 3 includes a web browser 350 and a browser plug-in module 
352 for interfacing web browser 350 with a main client module 360. Client module 360 
includes an event registry 362, video/audio decoder (s) 3 64, video /audio renderer(s) 
365, playout buffer(s) 366, and one or more dynamically loadable event applet(s), e.g., 
flipper applet 367, ticker applet 368 and VCR applet 369. In this embodiment, event 
registry 362 also functions as an annotation interpreter 363. 

Detailed Description Text (18) : 

The present invention is directed at the efficient and reliable streaming of data 
packets from stream server 220 to client computer 240, accomplished by optimally 
utilizing the bandwidth of the connection provided by computer network 2 90 while 
minimizing the loss of packets. In one embodiment, the transmission rate of the data 
stream is dynamically adjusted in response to changes in the bandwidth made available 
by computer network 2 90 for the network connection between server 22 0 and client 
computer 240. Accordingly, server 220, in response to feedback from client computer 
240, dynamically selects transmission rates in order to better match the varying 
bandwidth capacity of the network connection. For example, server 220 streams video 
packets at 1 frames/second (fps) , 5 fps, 10 fps, and 15 fps for bandwidths of 4 
kbits/second (kbps) , 14 kbps, 18 kbps, and 44 kbps. 

Other Reference Publication (4) : 

"Web Theater Product User Guide, Version 2.0", Palo Alto, CA: VXtreme, Inc., (1997). 
CLAIMS : 

4. The method of claim 1 wherein said data stream is a video data stream. 

12. The method of claim 7 wherein said data stream is a video data stream. 

18. The client computer of claim 15 wherein said data stream is a video data stream. 

26. The client computer of claim 21 wherein said data stream is a video data stream. 

32. The computer-readable medium of claim 29 wherein said data stream is a video data 
stream. 
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40. The computer-readable medium of claim 35 wherein said data stream is a video data 
stream. 
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TITLE: Audiovisual content distribution system 



Abstract Text (1) : 

A digital department system is disclosed. The digital department system of the present 
invention includes a network management center, a network operating center that is 
coupled to the network management center, a multimedia server, a multicasting 
transmission medium coupling the network operating center and the multimedia server, a 
listening post coupled to the multimedia server by the network, and one or more 
audio/video display capable of displaying video and capable of playing audio, the audio 
video display coupled to the multimedia server by the network. The audio and video 
content are stored in digitized files on the multimedia server for distribution 
throughout the site via the network. 

Application Filing Date (1) : 
19980807 

Brief Summary Text (13) : 

In one embodiment of the present invention, a digital department system is provided. 
The digital department system of the present invention includes a network management 
center, a network operating center that is coupled to the network management center, a 
multimedia server, a multicasting transmission medium coupling the network operating 
center and the multimedia server, a listening post coupled to the multimedia server by 
the network, and one or more audio/video display capable of displaying video and 
capable of playing audio, the audio video display coupled to the multimedia server by 
the network. The audio and video content are stored in digitized files on the 
multimedia server for distribution throughout the site via the network. 

Detailed Description Text (12) : 

NMC 110 maintains a catalog (database) of products for preview. NMC 110, in 
accumulating and compiling this information, also digitizes this information and 
provides it to a Network Operations Center (NOC) 120 in the form of digitized data 
files 122. It will be noted that data files 122, although referred to in terms of 
digitized audiovisual content, can also be streaming audio, streaming video, or other 
such information. Each product may have associated digital files containing information 
pertaining to the product. Alternatively, all the information may be compiled into one 
file. Following are examples of the types of files/information that may be catalogued 
and maintained: 

Detailed Description Text (75) : 

One example of a multicasting technique is the Multicast File Transfer Protocol (MFTP) 
from Starburst . TM . . This protocol is described in great detail in the specification 
entitle " STARBURST MULTICAST FILE TRANSFER PROTOCOL (MFTP) SPECIFICATION," (filename: 
draft-miller-mftp-spec-03.txt; dated April, 1998) which can be viewed at the time of 
this writing at the following universal resource locator on the World Wide Web : 

Detailed Description Text (89) : 

Receiver / decoder 13 5 is capable of receiving, processing, and providing voice, video, 
data, and other forms of information to various devices within commercial sales outlet 
130. While the configuration of the communications network 125 (and in particular, 
transmitting station 121, receiving station 128, and receiver/decoder 135) will vary 
according to the technology used to distribute digital data files 122. For example, 
while transmitting station 121 and receiving station 128 are described in terms of a 
satellite network, one of skill in the art will recognize that these elements could 
support broadcast or duplex communications systems. These elements could therefore be 
satellite transmitters/receiver pairs, a multicast network, a UUCP (Unix- to-Unix CoPy) 
network, or the like. Alternatively, these elements could be network interface cards, 
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microwave transceivers, infrared transceivers, or the like. In this example, a 
satellite broadcasting system is employed. Transmitting station 121 and satellite 127 
are implemented using commercially available satellite communication technology, and so 
are not described in detail herein. 

Detailed Description Text (129) : 

Receiver/decoder 13 5 is preferably capable of receiving both digital and analog 
information. With regard to data reception and distribution, receiver/decoder 135 is 
connected to various network nodes in commercial sales outlet 13 0 via a network system 
150. Network 150, while it may be directly connected to various other nodes in 
commercial sales outlet 130 is connected in FIG. IB to multimedia server 160, as noted. 
For most of data files 122, once they are received by receiving station 128 at 
commercial sales outlet 130, they are passed to receiver/ decoder 135, which in turn 
passes them on to multimedia server 160. Multimedia server 160 is connected via the 
network (e.g., an ether network using a TCP/IP protocol stack, and using FTP file 
transfers to distribute the promotional information) to various nodes in commercial 
sales outlet 130. These nodes include (but are not limited to) an in-line home video 
station 161, an in-line computer hardware and software interactive display 165, an 
in-store radio system 170, a wall-of-eyes 180 (which may be connected to either 
multimedia server 160, or to receiver / decoder 135 via a video switch 175, which is 
optional), listening posts 185, audio/video endcaps 190, audio/video endcaps in other 
departments 195, on-line services 200 and an entity LAN 210. Entity LAN 210 may be 
connected to an on-line internet commerce access system 220, a UNIX server 230 and/or a 
CBL server 240, among other such possible connections. 

Detailed Description Text (310) : 

As noted, data sent via communications network 12 5 can be of several types. For 
example, receiver /decoder 135 may provide audiovisual training information to video 
cassette recorder (VCR) 140 via the direct video capabilities that communications 
network 125 may support (e.g., real-time analog, high-definition television (HDTV), or 
digital video information) . Training VCR 140 allows such audiovisual information to be 
recorded for later playback, to allow the audiovisual information to be replayed for 
off-line training or during multiple training sessions. 
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** See image for Certificate of Correction ** 

TITLE: Remote semiconductor microscopy 

Application Filing Date (1) : 
19990423 

Detailed Description Text (11) : 

In yet another embodiment, the digitized frames of analog video 122 are streamed over 
the network 106 from the server 104 to the clients 102. In a further embodiment, the 
streaming video format can be the Advanced Streaming Format (ASF) (Microsoft . RTM . 
Corporation, Redmond, Wash.), further described in a document published by 
Microsoft .RTM. Corporation and Real Networks . TM. , Inc., entitled Advanced Streaming 
Format (ASF) Specification, Feb. 11, 1998, hereby incorporated by reference, and which 
may be found on the World Wide Web at http://www.microsoft.com/ asf /whitepr/asf wp . htm. 
Frames of digitized video data 122 are streamed in the ASF format by Netshow Server 
software operating on the server 104. The ASF video is played on the clients 102 by 
Netshow Player software. Netshow software is also a product of Microsoft . RTM . 
Corporation (Redmond, Wash.). However, the present invention can utilize other 
client-server streaming software, such as Real Video by Real Networks, Inc. (Seattle, 
Wash. ) . 

CLAIMS : 

2. The inspection system of claim 1 wherein the server streams the digitized video 
signal to the client computers. 

6. The inspection system of claim 1, wherein the video capture system includes an 
encoder-decoder (CODEC) to compress the digitized video signals. 

14. The semiconductor wafer microscopy system of claim 10, wherein the video capture 
system streams the digitized video signal to the client computer. 

23. The method of claim 22 wherein communicating the digitized video signal includes 
streaming the video signal. 

30. The computer -readable medium of claim 29 wherein communicating the digitized video 
signal includes streaming the video signal. 
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DOCUMENT -IDENTIFIER: US 6421726 Bl 

TITLE: System and method for selection and retrieval of diverse types of video data on 
a computer network 

Application Filing Date (1) : 
19980301 



Brief Summary Text (4) : 

The Internet, is growing, in terms of both size and sophistication, at a rapid rate. In 
the past, most users of the Internet were academic, research, or institutional users; 
the Internet was primarily used at that time to transmit and receive electronic mail 
and network news and to allow transfer of computer files. However, since the 
introduction of the World Wide Web (also known as the "Web" or the "WWW" ) several years 
ago, the Internet has begun to host increasing amounts of other types of data of 
general interest, namely representations of images, articles, etc. 

Brief Summary Text (5) : 

The Web protocol and language establish a graphical means to navigate the expanses of 
the Internet. " Web pages," often consisting primarily of text and graphical material, 
are stored on numerous computers, known as " Web servers," throughout the Internet. A 
software program known as a "browser" can be used to access and view Web pages across 
the Internet by specifying the location (i.e. Internet address) of the desired Web 
page. When a Web page is accessed, its information is transmitted from the remote 
computer (server or delivery site) , wherever in the world it may be located, across the 
Internet, to the user. 



Brief Summary Text (6) : 

In recent times, the Web has begun to host highly sophisticated types of multimedia 
content, such as audio and video data, and computer software. Compared to first 
generation Web content, namely text and still images, audio clips, video clips, and 
software programs have extremely high storage and bandwidth requirements. 

Brief Summary Text (7) : 

Before a video clip can be transmitted over a computer network, the clip must be 
digitized by encoding the video's analog signal to "Is" and "0s." In order to reduce 
the bandwidth required to transmit the digitized video, the video data stream is 
frequently compressed. Video compression is a process by which redundant data is 
eliminated from the video data stream so that the overall size of the data stream is 
reduced. There are many different compression formats which are used to reduce video 
data streams, e.g., MPEG, JPEG, H261, Indeo, Cinepak, AVI, Quicktime, TrueMotion and 
Wavelet . 



Brief Summary Text (8) : 

Video clips received in a compressed format generally s must be decompressed before 
they can be viewed. Decompression of a video is commonly performed by a video player 
"CODEC" program, or COmpressor/DECompressor , often located at a user's multimedia 
terminal. Generally speaking, a single CODEC program can only recognize and decompress 
a single compression format. 

Brief Summary Text (18) : 

As discussed above, a browser program can be used to access and view Web pages across 
the Internet by specifying the location (i.e. Internet address) of the desired Web 
page, or more commonly, by "hotlinking" to Web pages. Common browsers are Lynx, NCSA 
Mosaic, Netscape Navigator, and Microsoft Internet Explorer. The desired Web page is 
specified by a uniform resource locator ("URL"), indicating the precise location of the 
file using the syntax "http : //internet . address/directory/ filename. html" . 
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Brief Summary Text (19) : 

Web pages are generally described, in terms of layout and content, by way of a language 
known as "HTML" (HyperText Markup Language) . Any particular computer linked to the 
Internet can store one or more Web pages, i.e. computer files in HTML format, for 
access by users. 

Brief Summary Text (20) : 

Hotlinking from one HTML Web page to another is accomplished as follows. The user first 
accesses a Web page having a known address, often on the computer located at the user's 
ISP (Internet Service Provider) . The ISP is the organization providing Internet 
connectivity to the user. That Web page can contain, in addition to textual and visual 
data specified in HTML format, "links," or embedded information (in the form of URLs) 
pointing to the Internet addresses of other Web pages, often on other computers 
throughout the Internet. The user, by selecting a link (often by pointing and clicking 
with a mouse) , can then access other Web pages, which can in turn contain further data 
and/or additional links. 

Brief Summary Text (21) : 

Various extensions to HTML, such as Netscape's EMBED tag, allow references to other 
data to be embedded into Web pages. Some browsers are not capable of handling data 
other than text and images. Other browsers can handle the data in various ways. NCSA 
Mosaic, for example, handles references to unknown types of data by allowing the data 
to be downloaded to the user's computer, and then optionally invoking an external 
program to view or manipulate the data. Recent releases of Netscape Navigator and 
Microsoft Internet Explorer take the concept one step further: a browser extension, or 
"plug-in," can be automatically invoked to handle the data as it is received from the 
remote Web page. Other means, such as network program "applets" written in the Java 
language (or a similar language) , can be used to extend the functionality of the 
browser environment or network. 



Brief Summary Text (23) : 

The network design compromises discussed above generally adversely impact the 
transmission of audio and video data across the Internet. While a user using a browser 
to "surf" the Web might not notice minor delays and transmission rate variations while 
retrieving text and still images, such defects become apparent and significant when 
real-time audio and video information is accessed. 



Brief Summary Text (24) : 

In an attempt to solve these problems, Internet content providers sometimes spread 
popular content around the Internet on various servers or delivery sites known as 
"mirror sites." Each mirror site contains information that is essentially identical to 
that of the original site. For example, if a popular Web site is located in New York, 
mirror sites might be located in Los Angeles, London, and Tokyo. Accordingly, if a 
European user is having difficulty accessing the original New York site, he can hotlink 
to the mirror site that is geographically closest, i.e. London. 

Brief Summary Text (27) : 

Moreover, mirror sites are often hosted on a voluntary basis. If a Web site is 
extremely popular, and a service provider determines that the subject matter might be 
of interest to its subscribers, that service provider might agree to host a mirror site 
of the original Web site. Such an arrangement would be attractive to host of the mirror 
site because people would be drawn to the mirror site, and might hotlink to other 
content hosted there. On the other hand, such voluntary alliances typically are not 
reliable and might be severed at any time. 

Brief Summary Text (29) : 

Currently, there is no guidance in selecting optimal locations for delivery sites, nor 
is there a known method permitting a user to determine which mirror site to connect to 
that will ensure optimum performance. In fact, the use of a traditional mirror site is 
voluntary. Typically, a user will try to access the original site (or a known mirror 
site) , and will switch to another mirror site only if performance is found to be 
insufficient after one or more attempts. This approach is an inefficient utilization of 
network resources. Clearly, mirror sites are not an optimum solution to the problem of 
overloaded Web sites. A principal reason for this, among others, is the failure to 
consider network performance. 

Brief Summary Text (36) : 

The invention is directed to a system and method for the optimized distribution of Web 
content to and from sites located around the Internet. An intelligent mirroring scheme, 
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called here "Smart Mirroring," is used to determine the need for and distribution of 
mirror sites and to direct user requests for certain Web content to an optimum mirror 
site. A format selection scheme is used to automatically select a video clip file 
compressed in a format recognized by a CODEC program already installed and operative on 
a user terminal. 

Brief Summary Text (37) : 

A number of "smart" delivery or mirror sites are used to distribute popular Web content 
to various parts of the Internet. A comprehens ive scheme of network analysis, based on 
tests performed by a large number of users, is used to interactively determine the 
preferred locations for the sites, and to determine the optimum sites to be used by 
each individual user. 

Brief Summary Text (40) : 

A system according to the invention begins with an original Web site and at least one 
additional delivery (or mirror) site. Each user desiring to use the system will be 
provided, in a preferred embodiment, with software which includes a configuration 
utility and a client program. The configuration utility is used first to determine 
which delivery sites provide improved performance for that particular user. 

Brief Summary Text (42) : 

The configuration utility will run a subset of the tests specified in the delivery site 
file. The test results show which delivery sites yield improved performance for the 
user, and also contain information on various generalized network capabilities from the 
standpoint of the user running the tests. The network test results and the identity of 
the chosen delivery site will be sent (via e-mail in one possible configuration) back 
to the delivery service provider for incorporation into the service provider's 
database . 

Brief Summary Text (43) : 

The delivery site chosen by the configuration utility is then used by that user for the 
retrieval of all content managed by the delivery system service provider. Consequently, 
when the user is browsing Web content, and finds a particular item, e.g. a video clip, 
that is managed by the service provider's delivery system, the client software will 
automatically retrieve it from the specified "Smart Mirror" delivery site. Site 
preferences and default sites can be updated periodically on request, at specified 
times, or in response to changes in network load and traffic. 

Brief Summary Text (46) : 

A Web page hosting an embedded video clip also includes code to activate the Smart 
Mirroring and format selection schemes. Accordingly, a video clip selection and 
retrieval system according to the invention stores each video clip at various locations 
in a number of different compression formats. A format selection module comprises 
software programs that instruct the user terminal to determine which CODEC programs are 
installed at a user terminal, select a preferred CODEC for use, locate video clips 
stored in a compression format recognized by the preferred CODEC, and redirect or 
modify a video request so the video delivery system can locate and retrieve the desired 
clip. Where no appropriate CODEC is detected, the format selection module prompts the 
user to download and install a preferred CODEC. 

Brief Summary Text (47) : 

In a first embodiment of the format selection module, a video clip is stored in a 
number of different compression formats at individual delivery sites connected to the 
Internet. The video clip and its address are referenced on a Web page providing access 
to the server. The Web page also contains an embedded software program for querying the 
user terminal to determine what CODECs are installed at the terminal. The embedded 
program is automatically executed when the user requests a video clip which is 
referenced on the Web page. If the query detects the presence of a CODEC that can 
decompress a video clip stored by one of the servers, the embedded program will select 
that CODEC and request the appropriate video clip. 

Detailed Description Text (8) : 

For the purposes of describing this invention, a delivery site is a "node" on the 
network which may store data or other files, such as software code, for delivery. The 
term can also include a site which is responsible for data delivery, including mirror 
sites, content providers, and servers for broadcast video streams or Web sites. 

Detailed Description Text (17) : 

The configuration utility 34 then queries the user (step 42) for various items of 
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information needed in the configuration process, for example, the user's name, e-mail 
address, password, modem speed, and information related to access control (e.g. what 
levels of various attributes are viewable by the user) . The access control mechanism 
will be discussed in further detail below. In one embodiment of the invention, the 
information received from the user is encrypted and stored in a configuration file on 
the user terminal 12. 

Detailed Description Text (33) : 

In one embodiment of the invention, a proprietary graphical interface is provided so 
that the location of the user and the locations (both geographic and electronic) of 
each site tested can be displayed on a monitor connected to the user terminal 12, 
allowing a visual indication of the relative distances between sites. In one 
embodiment, the display is shown in the form of a "radar screen, " upon which the user 
terminal 12 and delivery sites are displayed as "blips" superimposed over a map of the 
pertinent geographical region. In order to encourage the user to use the application 
and to offer more network-wide data, the user interface can allow the user to enter an 
"ad-hoc" test site for additional performance testing. In this case, the configuration 
utility will test either the default Web page file (e.g., 11 index, html" ) or a specific 
file requested by the user. Analysis results from the user-selected site are adjusted 
so that reasonable comparisons can be made with results from other sites. 

Detailed Description Text (36) : 

After a Smart Mirror site is selected, certain data will be sent to the MSP 32 (step 
52) via e-mail or other Internet electronic protocol. The information received by 
querying the user, the identity of the selected Smart Mirror site, and all raw test 
data and results, including the time and date at which each test was run, is compiled 
into a text file (which is encrypted in one embodiment) . Upon receipt by the MSP 32, 
the data is stored in a database for use in managing and analyzing the system. 

Detailed Description Text (38) : 

It should be noted that in the operation of the system, the MSP 32 performs certain 
functions. The MSP 32 maintains the delivery site list, adding and deleting sites as 
necessary. The MSP 32 also maintains the database of network performance, containing 
information received via e-mail or other means from users running the configuration 
utility 34. As large amounts of data are received from numerous users, the database can 
provide valuable information on the performance and other characteristics of the 
Internet and portions thereof. Various data processing techniques are known to derive 
such information. 

Detailed Description Text (43) : 

In one embodiment of the invention, the Smart Mirror system is used to locate a 
delivery site from which to download a video or audio clip ("clip") referenced on a Web 
page. In this embodiment, the client program can be referred to or considered a "player 
program. " The player program, in addition to carrying out the functions of the client 
program 36, enables the retrieval and playback of video data. Ordinarily, a browser 
program 38 is run on the user terminal 12 to view Web content. Browser programs 
typically used include NCSA Mosaic, Netscape Navigator, and Microsoft Internet 
Explorer. The browser program 38 allows the user to hotlink among various Web sites on 
the Internet. 

Detailed Description Text (44) : 

The EMBED tag is used within HTML documents to indicate which Web pages include content 
managed by the system. When the browser program 38 receives a Web page containing an 
EMBED tag, a download of the file referenced by the tag is commenced, and the file type 
is analyzed. If the file is of a type handled by the player program 36, e.g. MPEG, the 
browser program 38 initiates the player program 36. The contents of the tag are then 
passed by the browser program 38 to the player program 36. 

Detailed Description Text (47) : 

If the player program 36 determines that the EMBED tag references a video clip or other 
content handled by the system (step 62), the transfer of the embedded clip from the 
content provider 22 is stopped. The player program 36 then extracts access control or 
rating information from the EMBED statement (step 64), if any exists. This rating 
information is compared against the reference levels stored in the configuration file 
stored at the user terminal 12 (step 66) . If rating information does not exist for the 
clip, the configuration file is queried to determine whether unrated clips, as defined 
below, may be played (step 68) . Based on the foregoing information, the player program 
36 will authorize or decline the viewing of the desired clip. 
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Detailed Description Text (52) : 

If the player program 3 6 determines that the EMBED tag references a video clip or other 
content not handled by the system, the player will check whether the access control 
level set in the configuration file allows the user to play these clips or other files 
which are considered "unrated" (step 92). If so, the clip is transferred from its 
original content provider 22 by traditional means (step 94) , and the player program 36 
displays the downloaded file (step 96) . If not, the player prevents the clip from being 
transferred (step 98) and displays a message (step 100) advising the user that the 
download is not permitted. 

Detailed Description Text (64) : 

A Web page embedding a reference to a video clip for use with the invention, as 
described in FIG. 4, also includes an embedded software program. This software program 
can be written in the JavaScript language, which is known in the art and compatible 
with a number of Web browsers presently available, including Netscape Navigator and 
Microsoft Internet Explorer. 

Detailed Description Text (65) : 

First, the Web page containing the video clip is loaded by the user terminal (step 
120) . The embedded software program "script" is then read and interpreted (step 122) by 
the browser. 



Detailed Description Text (75) : 

First the Web page containing the video clip is loaded by the user terminal (step 160) . 
The embedded software program "script" is then read and interpreted (step 161) . 

Detailed Description Text (84) : 

If the clip is downloaded directly (and the redirector features are not available) , the 
clip is played by traditional means. However, the format selection plug-in may have 
associated with it a configuration file, containing a listing of delivery sites 
containing the video clips managed by the invention. Instead of using the URL stored on 
the Web page for the download of a clip, the plug-in can make a random selection from 
the list of delivery sites, and append the name of the clip and the preferred format 
type to that selected delivery site. The download can then take place (step 194) from 
the randomly selected delivery site. While this does not accomplish the Smart Mirroring 
method, it does accomplish a certain level of load distribution. Although the choice 
made at a particular user terminal might not be optimum for that terminal, video clip 
demand will be spread over a plurality of delivery sites. 
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